Howard Kiewe talks to Harvard’s B Price Kerfoot about how revisiting learning over time can make it more sustainable
I recently attended the Society of Pharmaceutical and Biotech Trainers' conference in Orlando, Florida, which featured many interesting presentations, including one by Dr B Price Kerfoot.
Kerfoot has degrees from Princeton, Oxford and Harvard (in medicine and education). His Ivy League education, 43 research publications and position as associate professor in surgery at Harvard Medical School attest to his intelligence and work ethic. But of greater interest to me was his decision to focus his research on education, rather than medicine: a choice some of his surgical colleagues described as "career suicide" but which, by all appearances, has been a smart one.
Kerfoot's commitment to education led to ground-breaking research in spaced education. In a series of carefully controlled randomised trials, he demonstrated that spacing out online learning over time (as opposed to consuming it all at once or 'bolus') results in improved information retention, knowledge transfer to related topics, and learning efficiency1, on-the-job performance2 and identification and remediation ofunderperforming students3,4.
More recently, Kerfoot studied the combination of spaced education with certain game mechanics (rules intended to make games engaging). Here he found game-like spaced education to increase knowledge retention and test performance while being associated with improved course completion rates and motivation to participate in future spaced education programmes5,6.
In addition to a full academic career, Kerfoot is a board member for Qstream, which has developed learning technology based on his research.
Kiewe: Tell me how you arrived at the concept of spaced education. What got you interested?
Kerfoot: My career focus has been on applying clinical trial rigour to the evaluation of education and educational technology. The first trial that I did was an eye-opener7. This was a randomised trial of about 360 students, and we showed that the standard web-based teaching modules can be very helpful for increasing knowledge in the short term, but long-term retention was extremely poor. Searching for an explanation, I reviewed the psychology research literature and two main principles seemed to be of value.
First was the spacing effect that had been studied as far back as the 19th century by Ebbinghouse. This shows that, if you present information and then reinforce it over spaced intervals of time, it increases the acquisition of knowledge and encodes it in such a way that it's preferentially retained.
Second was the testing effect, which shows that the act of testing increases retention. Most people think that testing is like sticking a dipstick into the tank to see how much knowledge is in there. However, the retrieval required during testing can itself dramatically improve long-term retention.
We combined the two effects and, to date, we've done about 16 large randomised trials with specific research questions as to when they work, what's the methodology, what's the value of the spacing, and what's the value of the testing. Over time the folks at Harvard generated a system by which we could automatically email a question to folks, have them record an answer, and give them immediate feedback. This system has been used in much of our research. Then we modified it to include various game mechanics, for example where the interval of reinforcement over time varies based on whether an individual answered a question correctly or not.
Kiewe: Okay, so what would you say the bottom-line findings are when comparing spaced to conventional learning?
Kerfoot: First, that the normal 'bolus' or 'binge and purge' education is not effective for long-term retention. Second, that there should be an assumption that learning is forgotten over time, and steps need to be taken not only to teach the material initially but also to maintain it. Third, that the spacing effect and the testing effect are very strong tools for achieving greater long-term retention. And finally, that the system we have developed to harness the spacing and testing effects has been very effective.
Kiewe: Interesting. Is it fair to say that, if you look at the amount of study time a person invests in learning something, they' re going to get better retention from their time investment if it's spaced out?
Kerfoot: Yes, and there' s a couple of studies that we've done that specifically have looked at this. In one we gave identical histopathology content and identical time of exposure to about 700 urology residents8. They were randomised to either get all the content bolus over a three-week period or spaced over 16 weeks. Then we tested them over the following 30 weeks to plot the forgetting curves.
We found that the cramming works for short-term learning. If you need something immediately, cram it, and you'll get a spike in learning. But then we found that, even after 20 weeks, the interns' ability to perform the histopathology diagnosis was back down to the baseline. This suggests that spacing can dramatically improve long-term retention even when the amount of time spent learning is held constant.
Kiewe: And the idea of an initial cramming burst followed by regular reinforcement, what is that effective for?
Kerfoot: That's a great question, because many of the people we speak with still very much feel they have a bulk of material they need to shove down people's throats. And if you have a short time period in which to get people up to a high level of knowledge, the binge and purge, bolus presentation can be effective.
That said, once you've reached that higher level, you can pair that with the spacing and testing approach over time to maintain that level of expertise.
However, that's only my extrapolation based on the existing research. Although we have one study9 that showed changes in clinical practice behaviour increased when a three-day medical conference was followed by spaced learning, we do not have a study that directly addresses knowledge retention in a systematic way.
Kiewe: Okay, let's talk about gamification. We hear that term a lot. Let's begin with the definition. How do you define gamification?
Kerfoot: I actually avoid the term. I'm concerned that, when people think of gamification, there's a sense that you're infantilising the process. Wow, I'm gonna get a badge, yay! If you add badges, you're badge-ifying and then, if you add leader boards, you're leader-board-ifying, and so on. The challenge here is that we're doing serious training for business, medicine, or the health of patients. So I tend to avoid that moniker. However, I think the key is that we take specific game elements - game mechanics - and incorporate them.
Kiewe: What are the elements that you might borrow from game design?
Kerfoot: There are three that we've looked at: rewards, teams and leader boards.
Kiewe: Let's start with rewards; what do you mean by that?
Kerfoot: Gabe Zickerman, with his gamification focus, has put forward a hierarchy for extrinsic rewards called SAPS: Status, Access, Power and Stuff. Status is external recognition you can earn, and you wouldn't believe what some people will do to be number one on a leader board or elsewhere. In Abu Dhabi, somebody paid $14m to have a vanity license plate with the number one. Okay, that was before the financial crisis, but still! Access: for example gilt.com is a site that allows access to sales earlier for people who spent $10,000 or more in a given year. Power: we see this a lot in video games. Stuff is the lowest in the hierarchy. It can have value, but it's problematic. For one thing, stuff costs money and so does its distribution.
Daniel Paint at Princeton looked at how this sort of extrinsic motivation can undermine the intrinsic motivation for creative tasks. I have to say I don't necessarily agree with some of his extrapolations, but I do think he brings up a larger point that there isn't a linear correlation between greater extrinsic rewards and greater performance. And that, if you can focus on intrinsically valuable rewards for people but not necessarily monetarily valuable, I think you get more bang for your buck.
The key is to align these extrinsic rewards - Status, Access, Power and Stuff - with a learner's intrinsic motivations. So, it actually works to strengthen the intrinsic rather than undermine it. And it's a challenge; I'm still learning a great deal myself.
Kiewe: What would that look like? Give me an example of aligning intrinsic and extrinsic.
Kerfoot: I can give you an example of what it doesn't look like! We've just completed a randomised trial with medical residents out of UCLA, looking at training for healthcare quality, and the trial specifically looked at game elements versus no game elements. Healthcare quality training is freaking torture and the last thing the residents want to do. So, the question was how can we devise a system to deliver that healthcare quality content in a way they would engage with and participate in? and we looked to game mechanics to accomplish this.
Both groups got the same baseline iterative questions over time but participants in one group were assigned teams and given team rewards, they were given aliases, and their scores were placed on a leader board. It was very interesting to see that, as we looked at engagement, both groups started at the same level, but those who were in the team/leader-board group saw a dramatic improvement in engagement over time while the control engagement tailed off.
That said, we get the greatest bang for the buck with content that learners feel is of value and, if we use this methodology with content that they feel is going to help them be better sales people or doctors, we're aligning those two motivations.
Kiewe: What about the impact of teams?
Kerfoot: With regards to teams, there was a recent controlled trial by Kevin Volpp and his group at the University of Pennsylvania that looked at weight loss in obese individuals rewarded either as individuals or as teams10. They randomised into two groups: in one group, individuals that reached their weight loss goals during a month received $100; the other group had five people and, if they collectively met the weight loss target, each individual would then get $100. So, the same amount of money per individual just structured differently. They found that those people that had the group rewards had greater weight loss and it was maintained more effectively long-term.
Kiewe: So we've covered rewards and teams, what about leader boards?
Kerfoot: One study we did with Harvard Medical School students had them write freeform answers to ethical questions and had other study participants vote as to what answers they liked the best, with the results tracked on a leader board. But, the leader board that we provided actually had people's names on it, and we thought this would be quite motivating.
Well, it was motivating, but it was motivating in that they didn't want to participate because those who didn't perform well felt shamed.
And so, on one hand, we've spoken to companies and they're like, yes we want people's names there. We show sales results and we published them with everybody's name on it. So, we have a culture of this sort of accountability and we want to align with that. On the other hand, when I have done studies with nurses, and most recently with employees of VA Boston, we gave them aliases and put them in teams. This provides a safe environment and, if they miss a question, they don't feel ashamed. But they're still motivated to participate for their teams.
Kiewe: So they know if they're doing well and, if they're not doing well, nobody else knows.
Kerfoot: Exactly. Well, their supervisors do but it's not public knowledge, and they're not going to be shamed in front of their peers. So it's been an interesting balance. We're learning all the time as to what works well and what doesn't in different environments. Overall, we've been very pleased.
Kiewe: Interesting. That speaks to the importance of safety in leader boards. What can you tell us about designing leader boards to enhance a learner's motivation?
Kerfoot: I remember when I was in high school, and the kids in the 7-Eleven surrounded the Asteroids game machine and looked at the leader board. If your score was high enough, you could put three initials there, and I never made it on. But ABE had 14 million! I remember thinking oh my God, I can never reach that level. I don't have enough quarters and neither do my parents. So, in that context, the leader board was interesting but not at all motivating.
But you can present leader boards in a way that's motivating and relatively meaningful to the participants. You can show the top 100 or the top teams, but what's also very helpful is to see where you stand. If you're number 38, you want to see who the people or the aliases are, who are 37 and 36, and how many points you need to pass them, and do you know them, are they from your sales district or are they from that other nursing ward. So, rather than an absolute scale, we use a relative scale, which is much more motivating.
Kiewe: So you don't put the actual scores on the board?
Kerfoot: We do, and we allow participants to access all leader boards, but we initially show them where they are relative to their close peers, so their leader board becomes an incentivising force rather than a disincentivising experience like when I looked at ABE's 14 million Asteroid points.
Kiewe: So you don't want them to feel hopeless. You want them to feel they have a reasonable shot of improving.
Kerfoot: Exactly. It's been a lot of fine tuning as to how you structure this in order to get an effective learning experience. And, luckily, we've been very successful. The ethics study at Harvard didn't go too well, but we learned a lot from that.
Kiewe: So, maybe as a closing thought, if you consider professionals in the area of instructional design, e-learning, or mobile learning, what advice would you have for them based on the research that we've discussed today?
Kerfoot: It's an excellent question. We have found that the question-explanation nugget of information can be very effectively delivered over mobile devices and can engage learners in such a way that other passive products like PDF, PowerPoint or video cannot. If you can design around the question-explanation structure, you can get a rich, engaging, potentially social, team-based learning experience that has demonstrated results.
1 Kerfoot B P, Fu Y, Baker H, Connelly D, Ritchey M L, Genega E M “Online Spaced Education Generates Transfer and Improves Long-Term Retention of Diagnostic Skills: A Randomized Controlled Trial” American College of Surgeons (2010)
2 Kerfoot B P, Lawler E V, Sokolovskaya G, Gagnon D, Conlin P R “Durable Improvements in Prostate Cancer Screening from Online Spaced Education” American Journal of Preventive Medicine (2010)
3 Kerfoot B P, Shaffer K, McMahon G T, Baker H, Kirdar J, Kanter S, Corbett E C, Berkow R, Krupat E, Armstrong E G “Online ‘Spaced Education Progress-Testing’ of Students to Confront Two Upcoming Challenges to Medical Schools” Academic Medicine (2011)
4 Kerfoot B P, Baker H, Pangaro L, Agarwal K, Taffet G, Mechaber A J, Armstrong E G “An Online Spaced- Education Game to Teach and Assess Medical Students: A Multi- Institutional Prospective Trial” Academic Medicine (2012)
6 Kerfoot B P, Baker H “An Online Spaced-Education for Global Continuing Medical Education” Annals of Surgery (2012)
7 Kerfoot B P, Baker H, Jackson T L, Hulbert W C, Federman D D, Oates R D, DeWolf W C “A Multi-Institutional Randomized Controlled Trial of Adjuvant Web-Based Teaching to Medical Students” Academic Medicine (2006)
8 Kerfoot B P, Fu Y, Baker H, Connelly D, Ritchey M L, Genega E M “Online Spaced Education Generates Transfer and Improves Long-Term Retention of Diagnostic Skills: A Randomized Controlled Trial” American College of Surgeons (2010)
9 Shaw T, Long A, Chopra S, Kerfoot B P “Impact on Clinical Behavior of Face-To-Face Continuing Medical Education Blended with Online Spaced Education: a Randomized Controlled Trial” The Journal of Continuing Education in the Health Professions (2011)
10 Kullgren J T, Troxel A B, Loewenstein G, Asch D A, Norton L A, Wesby L, Tao Y, Zhu J, Volpp K G “Individual-versus Group-Based Financial Incentives for Weight Loss: a Randomized Controlled Trial” Annals of Internal Medicine (2013)
Chocolate and zinc: who knew?
Uninspiring elearning script? Never fear, Tanya Chopp has some good ideas.
Owen Rose looks to behavioural sciences and the advertising industry for inspiration in his training design.
Vincent Belliveau, Senior Vice President & General Manager EMEA at Cornerstone OnDemand, explores the benefits of internal recruitment