Correlation Does Not Equal Causation (No Matter What You Think)
I want to talk about correlation versus causation, and how some people are spinning the recent full release of the results of the National Assessment of Educational Progress (NAEP) testing, and I want to do it by proving that I have the power to control the destiny of my favorite professional basketball team.
Recently, I was watching the Chicago Bulls play the Boston Celtics in their fourth game of the young season.
As a lifelong Bulls fan who has a hard time keeping my emotional equilibrium while watching games, when the Bulls fell behind 35-16 with over three minutes left in the first quarter after playing some truly appalling defense, I snapped off the television and picked up a book I was supposed to be reading.
Half an hour or so later I checked the score and saw that somehow the Bulls had turned the table, leading 65-54 at halftime. Excited to pick things up in the second half, I again tuned in and watched Boston chip away at the lead until the Celtics’ Marcus Smart chucked in a deep three-pointer to narrow the gap to 69-66.
I turned the TV off again. This time I monitored the game on my phone, checking in on the score every couple of minutes. I was not going to turn that TV back on for anything, convinced that my watching the game was directly related to the performance of the team. The Bulls steadily expanded their lead, ultimately settling in for a 120-102 victory against one of the top teams in the Eastern Conference.
While I was watching the Bulls live, they were outscored 47-20. When I was not watching, they outscored the Celtics 100-55. Clearly, a key to the Bulls victory was me not watching the team.
You’re welcome, Bulls fans.
Obviously, I know that I am being irrational, that little old me has no control over the fate of the sports teams I favor, that I am not causing the Bulls to play poorly while I am watching them, but merely that there is a correlation between those two things. But once you have convinced yourself something is true, it is difficult to break free from your own, narrow point of view.
I’ve seen some similar responses to the NAEP scores, particularly among those folks who are convinced that the amount and duration of remote learning is the most powerful, or even sole cause of declines in those NAEP scores.
For example, Emily Oster, an economist at Brown University, and a go-to voice on Covid policies for many during the height of the pandemic took to Twitter to highlight some correlations between states with larger drops, and those which had longer periods of remote learning.
Her case, however, is weak, as there is a loose correlation for math scores, and no correlation for scores on reading. The case is even weaker if we recognize that correlation does not equal causation, and the data from the NAEP scores is insufficient to identify all of the factors that may be impacting student achievement on these tests.
For one, decisions to move back to in-person schooling were more likely to happen at the district, rather than the state level. This makes even the correlation data questionable.
For another, this says nothing about the many other things that were happening in the midst of this major once-in-a-lifetime (let’s hope!) disruption.
As Sarah Karp reports at WBEZ, the reality is that we simply don’t know how big a role remote learning as an independent variable played in the drop in scores.
Oster has been a big proponent that remote learning is the chief cause of “learning loss” and that this loss was unnecessary because children could have safely gone back to school earlier. When we can find a correlation that confirms something we already believe is true — like that fact that I have bad luck when it comes to rooting for Chicago sports teams — we’re more likely to invest that correlation with the power of causation.
As someone who called the previous reactions to the initial announcement of NAEP scores a “freakout,” because the rhetoric was so over the top, I have to be cautious about potentially discounting these correlations.
There’s no doubt that remote schooling as part of the overall disruption of the pandemic was hugely challenging for students, teachers, and families alike, but by settling on one (fairly weak) correlation as sufficient proof, we’re closing off the inquiries that will allow us to see the full scope of causes and solutions that will get students back on track.
My chief worry is that believing that the entirety (or at least the bulk) of the blame lies in remote schooling may create a sense of complacency, the sense that now that students are back in school, issues will resolve.
But this presumes that the pre-pandemic status quo was working, a dubious stance.
Watching the Bulls play the Indiana Pacers, I resolved to break my superstition, and for a dominant first half, I felt well on my way to success. And then the Bulls’ lead shrank to a mere four points in the third quarter as the Pacers sank nine consecutive three-pointers.
Thankfully, the Bulls re-expanded the lead and I could rest easy for at least half of the fourth quarter, but that didn’t stop me from repeating, “It’s not your fault” over and over in my head.