answer quizzes enhance later exam per for mance in middle and high school classes, Journal of Experimental Psychology: Applied (in press).

15. These studies examined students’ use of testing as a study strategy: J. D. Karpicke, A. C. Butler, & H. L. Roediger, III, Metacognitive strategies in student learning: Do students practice retrieval when they study on their own?, Memory 17

(2009), 471– 479, and N. Kornell & R. A. Bjork, The promise and perils of self regulated study, Psychonomic Bulletin & Review 14 (2007), 219– 224. These studies reported the surveys of college students’ use of retrieval practice as study technique.

16. Taking a test— even when one fails to correctly recall information on it— enhances learning from a new study episode. See K. M. Arnold & K. B. McDermott, Test- potentiated learning: Distinguishing between the direct and indirect effects of tests, Journal of Experimental Psychology: Learning, Memory and Cognition 39 (2013), 940– 945.

Notes to Pages 43–48 ê 263

17. This is a study of frequent low- stakes testing: F. C. Leeming, The exam- a-day procedure improves per for mance in psychology classes, Teaching of Psychology 29 (2002), 210– 212. The author found that in sections in which he gave students a short test at the start of every class the students attended class more often and felt that they studied more and learned more than students in classes with only four tests throughout the semester. Final test per for mance for the different sections (quiz a day or no quiz a day) confi rmed students’ impressions. Another interesting study conducted in a classroom is K. B. Lyle & N.

A. Crawford, Retrieving essential material at the end of lectures improves per for mance on statistics exams, Teaching of Psychology 38 (2011), 94– 97.

Two reviews of research on retrieval practice and testing appear in H. L. Roediger & J. D. Karpicke, The power of testing memory: Basic research and implications for educational practice, Perspectives on Psychological Science 1 (2006), 181– 210.

This paper represents a comprehensive review of laboratory and classroom studies over nearly one hundred years of research, showing that testing can be a powerful learning tool. A more recent review points to many benefi ts of frequent testing in addition to the direct benefi t from retrieval practice: H. L.

Roediger, M. A. Smith, & A. L. Putnam, Ten benefi ts of testing and their applications to educational practice, in J. Mestre & B. H. Ross (eds.), Psychology of Learning and Motivation (San Diego: Elsevier Academic Press, 2012). This chapter provides a summary of the host of potential benefi ts of using testing as a learning technique.

3. Mix Up Your Practice

1. The report of the beanbag study can be found in R. Kerr & B.

Booth, Specifi c and varied practice of motor skill, Perceptual and Motor Skills 46 (1978), 395– 401.

2. Many

well-

controlled experiments conducted with a variety of materials and training tasks provide solid evidence that massed practice (doing the same thing over and over repeatedly, a strategy often preferred by learners) is inferior to spacing and

Notes to Pages 48–53 ê 264

interleaving of practice for learning and retention. A review of the literature on the spacing effect in memory can be found in N. J. Cepeda, H. Pashler, E. Vul, J. T. Wixted, & D. Rohrer, Distributed practice in verbal recall tasks: A review and quantita-tive synthesis, Psychological Bulletin 132 (2006), 354– 380.

3. The surgery study is C-A. E. Moulton, A. Dubrowski, H. Mac-Rae, B. Graham, E. Grober, & R. Reznick, Teaching surgical skills: What kind of practice makes perfect?, Annals of Surgery 244 (2006), 400– 409. This study randomly assigned surgical residents to either a normal daylong intensive lesson on a surgical procedure or to an experimental lesson that spaced four short periods of instruction over several weeks. The fi ndings, showing better retention and application of the surgical techniques after spaced instruction, prompted the medical school to reexamine their standard instructional procedure of cramming instruction on a par tic u lar surgical technique into one intensive session.

4. The study showing the benefi t of interleaving in mathematics problems is D. Rohrer & K. Taylor, The shuffl ing of mathematics problems improves learning, Instructional Science 35

(2007), 481– 498. The standard practice in mathematics textbooks is to cluster practice problems by problem type. This laboratory experiment demonstrated that this standard practice produced inferior per for mance on a fi nal test in which new problems of each problem type were given relative to a practice procedure in which the practice problems from different problem types were shuffl ed (interleaved).

5. The study relating differences in practice strategies to differences in motor- memory consolidation was by S. S. Kantak, K.

J. Sullivan, B. E. Fisher, B. J. Knowlton, & C. J. Winstein, Neural substrates of motor memory consolidation depend on practice structure, Nature Neuroscience 13 (2010), 923– 925.

6. The anagram study was by M. K. Goode, L. Geraci, & H. L.

Roediger, Superiority of variable to repeated practice in transfer on anagram solution, Psychonomic Bulletin & Review 15

(2008), 662– 666. These researchers gave subjects practice on solving anagrams for a set of words: one group was given the same anagram for a par tic u lar target word on every practice

Notes to Pages 54–60 ê 265

trial (massed practice), whereas another group was given a different anagram for a par tic u lar target word on each practice trial (varied practice). Surprisingly, varied practice produced better per for mance on a fi nal trial in which the anagrams were the very ones that were practiced in the other group that had practiced the tested anagram repeatedly.

7. The study about learning of artists’ styles was by N. Kornell & R. A. Bjork, Learning concepts and categories: Is spacing the

“enemy of induction”?, Psychological Science 19 (2008), 585–

592. In these experiments, college students attempted to learn the painting style of a number of relatively unknown artists.

Students learned the styles better when the paintings of the artists were interleaved compared to when each artist’s paintings were massed during learning. Yet, at odds with the objective learning outcomes, most of the learners insisted that they learned better with the massed pre sen ta tions. Another informative study is S. H. K. Kang & H. Pashler, Learning painting styles: Spacing is advantageous when it promotes discriminative contrast, Applied Cognitive Psychology 26 (2012), 97– 103, which showed that mixing the examples of paintings helped to highlight the differences among paint ers’ styles (what we are calling discriminative contrast).

8. The fi nding that improving discrimination among examples contributes to conceptual learning is from L. L. Jacoby, C. N.

Wahlheim, & J. H. Coane, Test- enhanced learning of natural concepts: effects on recognition memory, classifi cation, and metacognition, Journal of Experimental Psychology: Learning, Memory, and Cognition 36 (2010), 1441– 1442.

9. Peter Brown interview of Doulas Larsen, December 23, 2011, St. Louis, MO. All quotes from Larsen are from this interview.

10. Doug Larsen’s work can be found in D. P. Larsen, A. C. Butler, & H. L. Roediger, Repeated testing improves long- term retention relative to repeated study: a randomized controlled trial. Medical Education 43 (2009), 1174– 1181; D. P. Larsen, A. C. Butler, A. L. Lawson, & H. L. Roediger, The importance of seeing the patient: Test- enhanced learning with standardized patients and written tests improves clinical application of knowledge, Advances in Health Science Education 18 (2012), 1– 17; and

Notes to Pages 62–69 ê 266

D. P. Larsen, A. C. Butler, & H. L. Roediger, Comparative effects of test- enhanced learning and self- explanation on long-term retention, Medical Education 47, 7 (2013), 674– 682.

11. Peter Brown interview of Vince Dooley, February 18, 2012, Athens, GA. All quotes of Dooley are from this interview.

12. Psychologists interested in learning have long distinguished between momentary per for mance and underlying learning (as mea sured after a delay with intervening reminders). As a simple example, someone might tell you that James Monroe was the fi fth US president. You would probably be able to answer correctly if asked about the fi fth president for the rest of the day or the week. That would be due to having just heard it (thus boosting the momentary strength or what the psychologists Robert and Elizabeth Bjork call retrieval strength). However, if someone asks you a year later about the fi fth president, this would be a mea sure of habit strength or, as the Bjorks call it, storage strength. See R. A. Bjork & E. L. Bjork, A new theory of disuse and an old theory of stimulus fl uctuation, in A. F.

Healy, S. M. Kosslyn, & R. M. Shiffrin (eds.), From learning pro cesses to cognitive pro cesses: Essays in honor of William K. Estes (vol. 2, pp. 35– 67) (Hillsdale, NJ: Erlbaum, 1992).

For a recent discussion, see N. C. Soderstrom & R. A. Bjork, Learning versus per for mance, in D. S. Dunn (ed.), Oxford Bibliographies online: Psychology (New York: Oxford University Press, 2013) doi 10. 1093/obo/9780199828340-0081.

4. Embrace Diffi culties

1. All quotes of Mia Blundetto are from telephone conversations between Peter Brown, in Austin, TX, and Blundetto, at Camp Fuji, Japan, on February 9 and March 2, 2013.

2. The phrase “desirable diffi culties in learning” originated in the article R. A. Bjork & E. L. Bjork, A new theory of disuse and an old theory of stimulus fl uctuation, in A. F. Healy, S. M.

Kosslyn, & R. M. Shiffrin (eds.), From learning pro cesses to cognitive pro cesses: Essays in honor of William K. Estes (vol.

2, pp. 35– 67) (Hillsdale, NJ: Erlbaum, 1992). The idea seems counterintuitive— how can making a task more diffi cult lead

Notes to Pages 73–78 ê 267

to it’s being learned better and retained longer? The rest of this chapter explains this puzzle and why it seems to arise.

3. Psychologists distinguish among three stages in the learning