percent of the study time in recitation.
A second landmark study, published in 1939, tested over three thousand sixth graders across Iowa. The kids studied six- hundred- word articles and then took tests at various times before a fi nal test two months later. The experiment showed a couple of interesting results: the longer the fi rst test was delayed, the greater the forgetting, and second, once a student
To Learn, Retrieve ê 31
had taken a test, the forgetting nearly stopped, and the student’s score on subsequent tests dropped very little.5
Around 1940, interest turned to the study of forgetting, and investigating the potential of testing as a form of retrieval practice and as a learning tool fell out of favor. So did the use of testing as a research tool: since testing interrupts forgetting, you can’t use it to mea sure forgetting because that “contami-nates” the subject.
Interest in the testing effect resurfaced in 1967 with the publication of a study showing that research subjects who were presented with lists of thirty- six words learned as much from repeated testing after initial exposure to the words as they did from repeated studying. These results— that testing led to as much learning as studying did— challenged the received wisdom, turned researchers’ attention back to the potential of testing as a learning tool, and stimulated a boomlet in testing research.
In 1978, researchers found that massed studying (cramming) leads to higher scores on an immediate test but results in faster forgetting compared to practicing retrieval. In a second test two days after an initial test, the crammers had forgotten 50 percent of what they had been able to recall on the initial test, while those who had spent the same period practicing retrieval instead of studying had forgotten only 13 percent of the information recalled initially.
A subsequent study was aimed at understanding what effect taking multiple tests would have on subjects’ long- term retention. Students heard a story that named sixty concrete objects. Those students who were tested immediately after exposure recalled 53 percent of the objects on this initial test but only 39 percent a week later. On the other hand, a group of students who learned the same material but were not tested at all until a week later recalled 28 percent. Thus, taking a single test boosted per for mance by 11 percent after a week.
Make It Stick ê 32
But what effect would three immediate tests have relative to one? Another group of students were tested three times after initial exposure and a week later they were able to recall 53
percent of the objects— the same as on the initial test for the group receiving one test. In effect, the group that received three tests had been “immunized” against forgetting, compared to the one- test group, and the one- test group remembered more than those who had received no test immediately following exposure. Thus, and in agreement with later research, multiple sessions of retrieval practice are generally better than one, especially if the test sessions are spaced out.6
In another study, researchers showed that simply asking a subject to fi ll in a word’s missing letters resulted in better memory of the word. Consider a list of word pairs. For a pair like foot- shoe, those who studied the pair intact had lower subsequent recall than those who studied the pair from a clue as obvious as foot- s_ _e. This experiment was a demonstration of what researchers call the “generation effect.” The modest effort required to generate the cued answer while studying the pairs strengthened memory of the target word tested later (shoe).
Interestingly, this study found that the ability to recall the word pair on later tests was greater if the practice retrieval was delayed by twenty intervening word pairs than when it came immediately after fi rst studying the pair.7 Why would that be?
One argument suggested that the greater effort required by the delayed recall solidifi ed the memory better. Researchers began to ask whether the schedule of testing mattered.
The answer is yes. When retrieval practice is spaced, allowing some forgetting to occur between tests, it leads to stronger long- term retention than when it is massed.
Researchers began looking for opportunities to take their inquiries out of the lab and into the classroom, using the kinds of materials students are required to learn in school.
To Learn, Retrieve ê 33
Studying the Testing Effect “In the Wild”
In 2005, we and our colleagues approached Roger Chamberlain, the principal of a middle school in nearby Columbia, Illinois, with a proposition. The positive effects of retrieval practice had been demonstrated many times in controlled laboratory settings but rarely in a regular classroom setting.
Would the principal, teachers, kids, and parents of Columbia Middle School be willing subjects in a study to see how the testing effect would work “in the wild”?
Chamberlain had concerns. If this was just about memorization, he wasn’t especially interested. His aim is to raise the school’s students to higher forms of learning— analysis, synthesis, and application, as he put it. And he was concerned about his teachers, an energetic faculty with curricula and varied instructional methods he was loath to disrupt. On the other hand, the study’s results could be instructive, and participation would bring enticements in the form of smart boards and
“clickers”— automated response systems—for the classrooms of participating teachers. Money for new technology is famously tight.
A sixth grade social studies teacher, Patrice Bain, was eager to give it a try. For the researchers, a chance to work in the classroom was compelling, and the school’s terms were accepted: the study would be minimally intrusive by fi tting within existing curricula, lesson plans, test formats, and teaching methods. The same textbooks would be used. The only difference in the class would be the introduction of occasional short quizzes. The study would run for three semesters (a year and a half), through several chapters of the social studies textbook, covering topics such as ancient Egypt, Mesopotamia, India, and China. The project was launched in 2006. It would prove to be a good decision.
Make It Stick ê 34
For the six social studies classes a research assistant, Pooja Agarwal, designed a series of quizzes that would test students on roughly one- third of the material covered by the teacher.
These quizzes were for “no stakes,” meaning that scores were not counted toward a grade. The teacher excused herself from the classroom for each quiz so as to remain unaware of which material was being tested. One quiz was given at the start of class, on material from assigned reading that hadn’t yet been discussed. A second was given at the end of class after the teacher had covered the material for the day’s lesson. And a review quiz was given twenty- four hours before each unit exam.
There was concern that if students tested better in the fi nal exam on material that had been quizzed than on material not quizzed, it could be argued that the simple act of reexposing them to the material in the quizzes was responsible for the superior learning, not the retrieval practice. To counter this possibility, some of the nonquizzed material was interspersed with the quiz material, provided as simple review statements, like “The Nile River has two major tributaries: the White Nile and the Blue Nile,” with no retrieval required. The facts were quizzed for some classes but just restudied for others.
The quizzes took only a few minutes of classroom time.
After the teacher stepped out of the room, Agarwal projected a series of slides onto the board at the front of the room and read them to the students. Each slide presented either a multiple choice question or a statement of fact. When the slide contained a question, students used clickers (handheld, cell-phone- like remotes) to indicate their answer choice: A, B, C, or D. When all had responded, the correct answer was revealed, so as to provide feedback and correct errors. (Although teachers were not present for these quizzes, under normal circumstances,
To Learn, Retrieve ê 35
with teachers administering quizzes, they would see immediately how well students are tracking the study material and use the results to guide further discussion or study.) Unit exams were the normal pencil- and- paper tests given by the teacher. Exams were also given at the end of the semester and at the end of the year. Students had been exposed to all of the material tested in these exams through the teacher’s normal classroom lessons, homework, worksheets, and so on, but they had also been quizzed three times on one- third of the material, and they had seen another third presented for additional study three times. The balance of the material was neither quizzed nor additionally reviewed in class beyond the initial lesson and what ever reading a student may have done.
The results were compelling: The kids scored a full grade level higher on the material that had been quizzed than on the material that had not been quizzed. Moreover, test results for the material that had been reviewed as statements of fact but not quizzed were no better than those for the nonreviewed material. Again, mere rereading does not much help.
In 2007, the research was extended to eighth grade science classes, covering ge ne tics, evolution, and anatomy. The regimen was the same, and the results equally impressive. At the end of three semesters, the eighth graders averaged 79 percent (C+) on the science material that had not been quizzed, compared to 92 percent (A−) on the material that had been quizzed.
The testing effect persisted eight months later at the end-of- year exams, confi rming what many laboratory studies have shown about the long- term benefi ts of retrieval practice. The effect doubtless would have been greater if the retrieval practice had continued and occurred once a month, say, in the intervening months.8
Make It Stick ê 36
The lesson from these studies has been taken to heart by many of the teachers at Columbia Middle School. Long after concluding their participation in the research studies, Patrice Bain’s sixth grade social studies classes continue today to follow a schedule of quizzes before lessons, quizzes after lessons, and then a review quiz prior to the chapter test. Jon Wehrenberg, an eighth grade history teacher who was not part of the research, has knitted retrieval practice into his classroom in many different forms, including quizzing, and he provides additional online tools at his website, like fl ashcards and games.
After reading passages on the history of slavery, for example, his students are asked to write down ten facts about slavery they hadn’t known before reading the passages. You don’t need electronic gadgetry to practice retrieval.
Seven sixth and seventh graders needing to improve their reading and comprehension skills sat in Michelle Spivey’s En-glish classroom one period recently with their reading books open to an amusing story. Each student was invited to read a paragraph aloud. Where a student stumbled, Miss Spivey had him try again. When he’d gotten it right, she probed the class to explain the meaning of the passage and what might have been going on in the characters’ minds. Retrieval and elaboration; again, no technology required.
Quizzes at Columbia Middle School are not onerous events.
Following completion of the research studies, students’ views were surveyed on this question. Sixty- four percent said the quizzing reduced their anxiety over unit exams, and 89 percent felt it increased learning. The kids expressed disappointment on days when clickers were not used, because the activity broke up the teacher’s lecture and proved enjoyable.
Principal Chamberlain, when asked what he thought the study results indicated, replied simply: “Retrieval practice has a signifi cant impact on kids’ learning. This is telling us that
To Learn, Retrieve ê 37
it’s valuable, and that teachers are well advised to incorporate it into their instructional technique.”9
Are similar effects found at a later age?
Andrew Sobel teaches a class in international po liti cal economics at Washington University in St. Louis, a lecture course populated by 160– 170 students, mostly freshmen and sopho-mores. Over a period of several years he noticed a growing problem with attendance. On any given day by midsemester, 25– 35 percent of the class would be absent, compared to earlier in the semester when maybe 10 percent would be absent.
The problem wasn’t unique to his class, he says. A lot of professors give students their PowerPoint slides, so the students just stop coming to class. Sobel fought back by withholding his slides, but by the end of the semester, many students stopped showing up anyway. The class syllabus included two big tests, a midterm and a fi nal. Looking for some way to leverage attendance, Sobel replaced the big tests with nine pop quizzes. Because the quizzes would determine the course grade and would be unannounced, students would be well advised to show up for class.
The results were distressing. Over the semester, a third or more of the students bailed out. “I really got hammered in the teaching reviews,” Sobel told us. “The kids hated it. If they didn’t do well on a quiz they dropped the course rather than get a bad grade in it. Of those who stayed, I got this bifurcation between those who actually showed up and did the work, and those who didn’t. I found myself handing out A-plusses, which I’d never given before, and more Cs than I’d ever given.”10
With so much pushback, he had little choice but to drop the experiment and reinstate the old format, lectures with a midterm and fi nal. A couple of years later, however, after hearing a
Make It Stick ê 38
pre sen ta tion about the learning benefi ts of testing, he added a third major test during the semester to see what effect it might have on his students’ learning. They did better, but not by as much as he’d hoped, and the attendance problems persisted.
