(Bloomberg View) -- The term “school voucher” -- a credit that a student can use for tuition -- means that government subsidizes private schools. This is attractive to many people. It offers lower-income students an escape from inadequate, failing public schools, while promising religious parents the chance to send their kids to parochial schools much more cheaply.
But critics of school vouchers -- me included -- have pointed to evidence showing that school choice doesn’t improve academic performance. Although charter schools -- which are new schools established with the specific purpose of improving teaching techniques, often for the most disadvantaged students -- have shown some modest but promising success in raising test scores, vouchers have generally disappointed.
A recent survey paper by Greg Forster of the Friedman Foundation for Educational Choice suggests otherwise. Forster examines a number of papers studying school-choice experiments, and says that most of them find evidence of positive effects. His analysis was cited approvingly by the Wall Street Journal editorial page as evidence that voucher opponents have been cherry-picking the evidence.
If true, that would be a bombshell. But a closer look shows that it is Forster who overstates the evidence in favor of vouchers.
Forster surveys 18 papers that use randomization to measure the impact of school choice on academic outcomes. In these studies, lotteries decide who gets a school voucher and who doesn’t. Because lotteries are random, the difference between those who win the lottery and those who lose must also be random, rather than due to effects like only accepting high-skilled students.
But Forster’s evaluation of these papers often differs sharply from the authors’ own. For example, he cites a 2013 study by Marianne Bitler, Thurston Domina, Emily Penner and Hillary Hoynes, examining a New York City program, as showing positive effects of vouchers. But here is what Bitler et al. have to say about their own findings:
Our findings suggest that the NYC voucher experiment had little effect across the distribution of student achievement, with the possible exception of small negative effects in math at the top of the distribution.
To me, that doesn’t sound like the authors found a positive effect -- if anything, quite the opposite. But Forster marks this paper as a win for vouchers. Why? Because in an appendix, the authors replicate the method of a different paper, by William Howell and Paul Peterson, that found a positive effect of vouchers for some students. Forster declares that the methodology of the other paper is superior to that of Bitler et al., and thus views the replication as Bitler at al.’s real result, ignoring the authors’ own conclusions.
By doing this, Forster double-counts Howell and Peterson’s pro-voucher study, which is also included in his list of studies. And he ignores Bitler et al.’s own finding of a negligible or negative effect. This isn’t a good way to conduct a meta-analysis.
Another supposedly positive example Forster cites is a 2013 study by Patrick Wolf et al. of Washington, D.C.’s voucher program. The study indeed does show that vouchers increased graduation rates, which is good. But when it comes to test scores, a report on the same program by Wolf et al. for the U.S. Department of Education found the following:
There is no conclusive evidence that the [D.C. voucher program] affected student achievement. On average, after at least four years students who were offered [or used vouchers] had reading and math test scores that were statistically similar to [students who didn’t get voucher offers].
So vouchers again show no ability to raise test scores. By lumping together graduation rates and test scores, Forster muddles the fact that test scores themselves -- which are the outcome that most people argue about -- show negligible movement.
So Forster seems to have represented several studies as being more pro-voucher than they really are. Just as problematic, the studies that Forster leaves out are as important as the ones he chooses to include. Because his survey specifically excluded public-school choice programs, he doesn't mention a well-known 2006 study by Julie Cullen, Brian Jacob, and Steven Levitt. Those authors studied a public school-choice lottery program in Chicago and found no evidence that winning the lottery conferred any academic benefit. Academics who study education policy, such as Harvard economist Roland Fryer, are well aware of that result.
Forster’s report contains much more than this, but by now warning signs should be flashing. Perhaps this is unsurprising, given that Forster works for a think tank that defines its mission thusly:
The Friedman Foundation for Educational Choice is…solely dedicated to advancing Milton and Rose Friedman’s vision of school choice for all children…The Foundation promotes school choice as the most effective and equitable way to improve the quality of K–12 education in America.
A foundation dedicated to advancing a policy doesn’t seem to me like an unbiased source regarding the effectiveness of that policy. Of course, although voucher opponents may cherry-pick their evidence, voucher proponents clearly often do the same.
The issue of vouchers is critical at this time in light of their endorsement by the Trump administration, and more objectivity and less advocacy is needed if the country is going to make sound decisions. On one hand, there is a good argument for looking at vouchers’ effects not just on test scores, but on graduation rates, good behavior and parental satisfaction. It’s here that the evidence on vouchers is more encouraging. But on the other hand, a level-headed analysis of the issue would ask whether advantages like this can persist when all students are using vouchers, rather than just a few.
We should also take into account the costs of implementing a voucher system, which would involve major changes in public education, one of the U.S.’s core national institutions. Many families bought houses specifically to be near to good schools; a voucher program might change the quality of those schools suddenly and dramatically.
So this is a complex issue, and ideologically driven pro-voucher efforts don’t help to clarify the costs and benefits.
This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.
Noah Smith is a Bloomberg View columnist. He was an assistant professor of finance at Stony Brook University, and he blogs at Noahpinion.
To contact the author of this story: Noah Smith at nsmith150@bloomberg.net.
To contact the editor responsible for this story: James Greiff at jgreiff@bloomberg.net.
For more columns from Bloomberg View, visit http://www.bloomberg.com/view.