Getting Beyond Test Scores with Jack Schneider

Getting Beyond Test Scores with Jack Schneider

When I read Jack Schneider’s Beyond Test Scores: A Better Way to Measure School Quality, I nearly sprained my neck nodding in recognition and agreement. I was in the midst of writing about the ways standardized tests were specifically a barrier to helping students improve as writers, and here was an honest-to-goodness higher education expert who was tackling the bigger picture, showing how these same tests were a problem when it came to assessing schools.

I believe the discussion around whether or not standardized tests are “good” or “bad” would benefit from much more discussion about what kind of tests we’re talking about when we say “standardized tests,” when those tests are used, and what the results of those tests are used for. 

Prof. Schneider wrote Beyond Test Scores specifically in the context of his hometown of Boston, but the issues of how we judge schools and students are universal. He was kind enough to answer some questions over email.

John Warner: You’re a professor of education, but your interest in how we measure school quality came from a more direct personal experience. What happened? Why did this become the thread that you decided to pull on?

Jack Schneider: This line of research for me all stemmed from a moment of intense outrage. Our local paper, the Boston Globe, had rated schools, and the message was essentially: you’ve made a bad choice. I knew enough not to put much stock in such ratings. But it also just seemed so blatantly uncreative and ungenerous. The top-ranking schools were in the wealthiest white suburbs—confirming what lots of people already think. Socioeconomically diverse communities, like ours, which I frankly think are better for kids and better for America, were essentially doomed from the start. And, of course, rounding out the bottom of the list were the racially segregated schools serving Black and brown students. What possible value could such a ranking be adding? Why do it? But the moment of outrage was when, already incensed by the potential impact, I dug into the methodology. It was like, “Wait…it’s both morally wrong and methodologically stupid?” 


JW: What was wrong with the way the Boston Globe was rating your community’s schools?

JS: Imagine a world in which everything I can tell you about the quality of a school is rooted in a machine-scored standardized test. “It’s a good school, John. The kids fill out the right bubbles more often than the kids at other schools do. So you should definitely buy a home in the area.” It’s pretty silly. But now imagine an even wilder world in which the one thing I can tell you—the average results on these tests—is actually more a reflection of family background than school quality. We know from research that about two-thirds of student performance on standardized tests is predicted by variables like parental education. Only about one-third can be accounted for by schools. So what does the Boston Globe know about my daughter’s school? It basically knows the demography. It’s essentially saying: “Let’s rank schools by social and economic privilege and see what happens.”


JW: Rankings systems are always going to be a reflection of a particular set of values, and what struck me about your exploration of the Boston system in Beyond Test Scores is how very narrow and shallow the values underlying the ratings are. They actually don’t tell us much of anything. 

JS: Sit down with one of these rankings and ask the questions you really have. Not the status-anxiety questions. But the real questions. How safe are kids at this school? No idea. What are their relationships with teachers like? Haven’t the foggiest. Are there opportunities for creative play? Don’t know. What about music and the arts? Can’t tell you. Are they engaged in class? Not in the data. Is the curriculum rich and diverse? Not sure. Are kids happy here? Ask a Magic 8-Ball. These ratings are chiefly about selling advertisements. They have no idea what the schools are like, but they know quite well how to prey on the anxieties of parents. 


JW: What is your advice for parents about figuring out what matters to them and then assessing how their values fit with a particular school?

JS: It’s all embedded in your question. You already answered it! Start figuring out what matters to you. Don’t start by asking, “Where are the best schools?” That’s a trap. What does “best” mean? And are they actually measuring schools? You’re likely just going to learn which enclave of privileged kids has the most test-aligned curriculum. And hey, if that’s your heart’s desire—if you start with that as your aim—go ahead. I disagree with you, but go ahead. I just think that’s not where most of us will start. I think we’re going to start with: “I want my kid to feel known and cared for.” And we’re going to expand pretty broadly from there, because schools are places where we make the future for our kids. We want a lot from them, and almost none of that is captured by rankings and ratings. I’ve got a whole checklist that people can use, which is in my book. But ultimately, it’s about having a little faith that the kind of community you want to live in is going to have schools that reflect the community.


JW: I have written ad nauseum about my view that standardized tests, particularly as used for assessing student writing, are some combination of useless and actively destructive, but is there anything that standardized tests are useful for?

JS: Sure. But we have to stop acting like they measure school quality in order for that to be the case. If we did that, and we realized that they’re actually just telling us which kids need additional support, they might be pretty useful. Let’s look at the schools with lower scores and shower them with additional resources. That actually makes sense to me. These schools are often just as strong as their higher-scoring counterparts—in terms of teacher quality, the curriculum, the day-to-day functioning of the school. They’re just working with kids who need a little bit more support. 


Jack Schneider is an Assistant Professor in the College of Education at the University of Massachusetts Lowell and leads the Beyond Test Scores project. An award-winning scholar and teacher, his work explores the influence of history, culture, and rhetoric in education policy. The author of four books, Dr. Schneider has explored why particular ideas gain policy traction, how public perceptions of schools take shape, why education reform so often fails, and how organizations can use data to empower stakeholders. He is a co-founder of the Massachusetts Consortium for Innovative Education Assessment, co-editor of History of Education Quarterly, and co-host of the education policy podcast Have You Heard. In addition to his scholarly work, Dr. Schneider writes frequently in outlets like the Atlantic, the New York Times, and the Washington Post.