Bloom’s taxonomy of learning. To remove some of the abstraction from Bloom’s taxonomy, Wenderoth has translated her class material into the different levels of the taxonomy on an answer key to her tests. That is, for any given question, she provides a different answer for each level of the taxonomy: one that refl ects learning at the level of knowledge, a more thorough answer that refl ects understanding, a yet more complex answer that refl ects analysis, and so on. When students get their tests back, they also receive the answer key and are asked to identify where their answers fell on the taxonomy and to think about what they need to know in order to respond at a higher level of learning.
Closing the achievement gap in the sciences. Wenderoth and her colleagues have experimented with class structure and the principles of active learning to help close the achievement gap in the sciences. Poorly prepared students seldom survive entry-
Make It Stick ê 233
level science courses. As a result, even students whose interests and aptitudes might lead them to successful science careers never get through the door. For what ever reason, these students do not have a history from high school or family life of learning how to succeed in these highly challenging academic settings.
“For most of us who have found our way in the sciences,”
Wenderoth says, “any time we fell, there was somebody around to help us up, or to say, ‘This is how you get up.’ You were taught that when things don’t go well, you keep working anyway. You persevere.”
In their experiments, Wenderoth and her colleagues have compared the results of “low- structure” classes (traditional lecturing and high- stakes midterm and fi nal exams) with “high-structure” classes (daily and weekly low- stakes exercises to provide constant practice in the analytical skills necessary to do well on exams). They also teach students the importance of having a “growth mindset” (see the work of Carol Dweck, discussed in Chapter 7)— that is, that learning is hard work and that struggle increases intellectual abilities.
The results? High- structure classes in a gateway biology course signifi cantly reduced student failure rates compared to low- structure classes— narrowing the gap between poorly prepared students and their better prepared peers while at the same time showing exam results at higher levels on Bloom’s taxonomy. Moreover, it’s not just whether the student completes the practice exercises that matters. In the classes where exercises count toward the course grade, even at very low stakes, students achieve higher success over the course of the term compared to students in classes where the exercises are the same but carry no consequences for the grade.
“We talk to the students about how these are the habits of mind,” Wenderoth says. “This is the discipline that you have to
Make It Stick ê 234
have in order to succeed in the sciences. They’ve never thought about that, that every discipline has a culture. We teach them to think like the professionals they want to become. And when they fall, we show them how to get up again.”7
Michael D. Matthews, Psychology Professor,U.S. Military Academy at West Point The pedagogical philosophy at West Point is founded on an instructional system called the Thayer method, developed almost two hundred years ago by an early superintendent of the academy named Sylvanus Thayer. The method provides very specifi c learning objectives for every course, puts the responsibility for meeting those objectives on the student, and incorporates quizzing and recitation in every class meeting.
Students’ grades at the academy rest on three pillars of training: academic, military, and physical. Mike Matthews, a professor of engineering psychology at the academy, says the load on students is enormous, greater than the hours available to them. In order to survive at the academy, West Point cadets must develop an ability to zero in on what’s essential and let the rest fall by the wayside. “This is about having very high expectations across multiple dimensions and keeping them real busy,” Matthews, says. In fact, as stunning as it sounds, Matthews will tell a student, “If you’ve read every word of this chapter, you’re not being very effi cient.” The point is not to “slide your eyes over the words.” You start with questions, and you read for answers.8
There’s little or no lecturing in Matthews’s courses. Class opens with a quiz on the learning objectives from the assigned reading. From there, on many days, students “take to the boards.” The classrooms have slate on all four walls, and a group of students are sent to each blackboard to collaborate
Make It Stick ê 235
on answering a question given by the professor. These are higher- order questions than are given in the daily quiz, requiring the students to integrate ideas from the reading and apply them at a conceptual level. It’s a form of retrieval practice, generation, and peer instruction. One student is selected from each group to give a recitation to the class explaining how the group has answered the question, and then the group’s work is critiqued. All class meetings focus on constructs, not specifi c facts, and on the days the students do not take to the boards, they are engaged in other forms of exercise, demonstration, or group work aimed at understanding and articulat-ing the larger concepts underlying the matter at hand.
Clear learning objectives prior to each class, coupled with daily quizzing and active problem solving with feedback, keep students focused, awake, and working hard.
One of the most important skills taught at West Point is something learned outside the classroom: how to shoot an azimuth. It’s a skill used for keeping your bearings in unfamiliar territory. You climb a tree or a height of land and sight a distant landmark in the direction you’re headed. Compass in hand, you note how many degrees your landmark lies off of due north. Then you descend into the bush and keep working your way in that direction. Periodically, you pause to shoot an azimuth and make sure you’re on course. Quizzing is a way of shooting an azimuth in the classroom: are you gaining the mastery you need to get where you’re trying to go?
Matthews has had the privilege of seeing two of his students win Rhodes Scholarships. The most recent was Cadet Kiley Hunkler (now Second Lieutenant Hunkler). Hunkler will be spending the next two years at Oxford University, and then matriculating at Johns Hopkins Medical School. It was Hunkler who spoke to us of shooting an azimuth. “Everything at the academy is about self- responsibility, taking own ership for
Make It Stick ê 236
fi nding your own way to the objective,” she said.9 The Medical College Admission Test, for example, encompasses four major course blocks: reading, chemistry, physiology, and writing. For each of these blocks, Hunkler created the learning objectives in her head that she deemed most important and then set out to answer them as she studied. “I took a practice test every three days, saw what I got wrong, and adjusted.” Shooting her azimuth. “A lot of students get hung up studying for months, trying to memorize everything, but for me it was more about understanding the concepts. So my azimuth check would be, Okay, what is this question asking, what’s the broader theme here, and does that match up with what I’ve outlined for this section.”
One of this book’s authors (Roediger) attended Riverside Military Academy in Gainesville, Georgia, for high school.
Riverside used a form of the Thayer method, with students having daily quizzes, problem sets, or assignments to be completed in class. The range of ability of these younger cadets was much more varied than at the elite US Military Academy at West Point, but the Thayer method worked well. In fact, such methods that include daily participation are especially likely to help students who are not prone to work hard on their own outside of class. The Thayer method is a strong encouragement for them to keep at it, and echoes what Mary Pat Wenderoth (above) has found in her empirical studies: that high- structure classes help students who lack a history of using effective learning techniques and habits to develop them and succeed in rigorous settings.
Kathleen McDermott, Psychology Professor,Washington University at St. Louis Kathleen McDermott administers daily low- stakes quizzes in a university course on human learning and memory. It’s a class
Make It Stick ê 237
of twenty- fi ve students that meets twice a week for fourteen weeks, minus midterms and a fi nal exam. She gives a four-item quiz in the last three to fi ve minutes of every class. The questions hit the high points of the lecture, the readings, or both. If students have understood the material, they will get all four answers right, but they’ll have to think in order to do it. Anything covered in the course to date is fair game for a quiz, and she will sometimes draw from past material that she feels the students haven’t fully grasped and need to review.
McDermott sets the ground rules very clearly at the start of the term. She lays out the research on learning and the testing effect and explains why the quizzes are helpful, even if they don’t feel helpful. Students are allowed to drop four quizzes across the semester. In exchange, absences need not be justi-fi ed, and no missed quizzes will be made up.
Students initially are not happy about the quiz regime, and in the fi rst few weeks of the term McDermott will get email from students explaining why they had a legitimate excuse for an absence and should be allowed to make up a missed quiz.
She reiterates the terms: four free absences, no makeups.
McDermott says the quizzes provide an incentive for students to attend class and give students a way to contribute to their grade on a daily basis if they answer four out of four questions correctly. By the end of the semester, her students say that the quizzes have helped them keep up with the course and discover when they are getting off track and need to bone up.
“The key with quizzes is to establish very clear ground rules for the student, and make them manageable for the professor,” McDermott says. “As a student, you’re either there and you take it, or you’re not. For the professor, no hassling over makeup tests.”10
The quizzes in totality count for 20 percent of a student’s grade in the course. In addition, McDermott gives two
Make It Stick ê 238
midterm exams and a fi nal. The last two exams are cumulative. Having cumulative exams reinforces learning by requiring students to engage in spaced review.
Columbia, Illinois, Public School District As recounted in Chapter 2, we have worked with teachers in a middle school in Columbia, Illinois, to test the effects of integrating low-
stakes quizzing into the curriculum. Regular quizzing and other forms of retrieval practice have been adopted by teachers in the school who were a part of the research study and by others who were not but who observed the benefi cial results. The initial research project has since been extended into history and science classes in the district’s high school, where frequent retrieval practice is being used both to bolster learning and to help teachers focus instruction on areas where student understanding and per for mance need to be improved.
The Illinois State Board of Education has adopted new math and En glish language arts standards for K– 12 education in line with the Common Core State Standards Initiative led by the National Governors Association and endorsed by the nation’s secretary of education. Common Core establishes standards for college and career readiness that students should be able to meet on graduation from high school. The Columbia School District, like others, is redesigning its curriculum and its tests to be more rigorous and to engage students in more writing and analysis work, with the aim of promoting the higher- level skills of conceptual understanding, reasoning, and problem solving that will enable students to meet the standards established by the state. As one example of this overhaul, the sciences curriculum is being vertically aligned so that students are reexposed to a subject at various stages of their school
Make It Stick ê 239
careers. The result is more spaced and interleaved instruction.
In physical sciences, for instance, middle school students may learn to identify the six basic machines (inclined plane, wedge, screw, lever, wheel and axle, and pulley) and how they work, and then may return to these concepts in subsequent grades, delving into the underlying physics and how these basic tools can be combined and applied to solve different problems.
Tips for Trainers
Here are some ways trainers are using the same principles as those who teach in schools, in a variety of less structured and nonclassroom settings.
In- Service Training
Licensed professionals in many fi elds must earn continuing education credits to keep their skills current and maintain their licenses. As the pediatric neurologist Doug Larsen describes in Chapter 3, this kind of training for doctors is typically compressed into a weekend symposium, out of respect for participants’ busy schedules, set at a hotel or resort, and structured around meals and PowerPoint lectures. In other words, the strategies of retrieval practice, spacing, and interleaving are nowhere to be seen. Participants are lucky to retain much of what they learn.
If you see yourself in this scenario, there are a few things you might consider doing. One, get a copy of the pre sen ta tion materials and use them to quiz yourself on the key ideas, much as Nathaniel Fuller quizzes himself on the arc of a play, his lines, the many layers of character. Two, schedule follow- up emails to appear in your inbox every month or so with questions that require you to retrieve the critical learning you gained from the seminar. Three, contact your professional association
Make It Stick ê 240
and ask them to consider revamping their approach to training along the lines outlined in this book.
The testing effect forms the basis of a new commercial training platform called Qstream that helps trainers send learners periodic quizzes via their mobile devices to strengthen learning through spaced retrieval practice. Similarly, an emerging platform called Osmosis uses mobile and Web based soft-ware to provide learners access to thousands of crowdsourced practice questions and explanations. Osmosis combines the testing effect, spacing, and social networking to facilitate what its developers call “student- driven social learning.” Qstream (qstream.com) and Osmosis (osmose- it.com) suggest interesting possibilities for redesigning in-service training for
professionals. Many other companies are developing similar programs.
Kathy Maixner, Business Coach
