November 27, 2017
An Epidemiological Take on "Laptops Are Great. but Not During a Lecture or a Meeting." and Other Thoughts
Susan Dynarski in the New York Times, on students using laptops to take notes:
Measuring the effect of laptops on learning is tough. One problem is that students don’t all use laptops the same way. It might be that dedicated students, who tend to earn high grades, use them more frequently in classes. It might be that the most distracted students turn to their laptops whenever they are bored. In any case, a simple comparison of performance may confuse the effect of laptops with the characteristics of the students who choose to use them. Researchers call this “selection bias.”
It is true that measuring the effect of laptops on learning is difficult, but this is due to confounding, not selection bias. (Selection bias is “distortions that result from procedures used to select subjects and from factors that influence study participation”, according to Rothman (p. 134).) This is not merely a pedantic argument about terminology (though epidemiologists certainly love those) – it informs how you adjust the study design to compensate.
In a series of experiments at Princeton University and the University of California, Los Angeles, students were randomly assigned either laptops or pen and paper for note-taking at a lecture. Those who had used laptops had substantially worse understanding of the lecture, as measured by a standardized test, than those who did not.
This is the obvious methodology to control for confounding, but it misses something important: students may be choosing laptops over handwritten notes for a reason – perhaps they have physical trouble writing quickly and legibly, which makes handwriting more distracting than typing.
This is where drawing a hypothetical diagram of the underlying process can be helpful:
┌─────────────────┐
│ Choice to use │
┌────────▶ laptop or │────┐
│ │ handwrite notes │ │
│ └─────────────────┘ │
┌──────────────────┐ │ │ ┌────────────────────┐
│ True optimal │ │ │ │ │
│note-taking method│────────┴───────────────────────────────┴───▶ Learning outcomes │
│ │ │ │
└──────────────────┘ └────────────────────┘
In this case, I think something like the above is going on: students have a true optimal method for taking notes, which almost certainly will differ among individuals. They then make a choice about how to take notes. What the researchers really want to assess is how a non-optimal method of note-taking affects learning outcomes: for some students, the non-optimal method may be handwriting or it may be laptops.
The ideal study design would be to somehow determine the optimal note-taking method for each student, and then randomize to optimal or non-optimal. I would stratify the analysis into an “optimal=handwriting” and an “optimal=technology” group. (Though this misses the nuance of writing on a tablet using something like the Apple Pencil – see below.) How the student uses the note-taking method should also be controlled for, whether it’s doodling in the margins of paper notes or watching YouTube videos.
It would also be necessary to control for the professor’s lecture style and the subject area. And the assessment of learning outcomes would have to be standardized – simply relying on letter grades may introduce bias due to differences in assessment techniques among classes.
So yes, this is a very difficult thing to study, and I would encourage anyone trying to do so to consult with a methods expert ahead of time.
Dr. Dynarski continues:
Some students were told to perform small tasks on their laptops unrelated to the lecture, like looking up movie times. As expected, these students retained less of the lecture material. But what is really interesting is that the learning of students seated near the laptop users was also negatively affected.
This is absolutely true based on my own personal experience. There is a spectrum of laptop use from “only Microsoft Word is open and I’m only taking notes” to “I’m on Facebook for the entire lecture”. I would argue that any movement whatsoever into the latter group will have a large negative impact on learning outcomes because of the cognitive cost of context switching. And I am clearly evidence of the negative externalities of distracted laptop use because I was watching what other people were doing on their computers rather than paying attention to the lecture.
She concludes by writing:
…I find the evidence sufficiently compelling that I’ve made my decision: I ban electronics in my own classes.
I do make one major exception. Students with learning disabilities may use electronics in order to participate in class.
This only works if the professor has modified their lecture to work well with handwritten notes. I have taken many classes that rely heavily on PowerPoint slides and are impossible to keep up with as a student without using a computer or printing out the slides ahead of time.
For these classes, my favorite way to take notes was getting the slides ahead of time and then writing on top of them using an iPad. (You can do the same with slides printed on paper, but the iPad has many advantages like a color screen, zooming, multiple digital ink colors, and automatic handwriting conversion to searchable text.) If my professor was using PowerPoint to lecture, I would insist on this method of note-taking.
But what’s missing here is a discussion of the nontraditional lecturing methods that technology enables, like the flipped classroom. I would argue that fundamental changes to how courses are taught could have a much larger impact on learning outcomes than how students are taking their notes.
Comments? Please send me a message.