No Magic Bullet: Yet Another New Report Finds Uneven Effects for Charters

No Magic Bullet: Yet Another New Report Finds Uneven Effects for Charters: "
Yet another report has just been released which shows that charters have extremely uneven effects on student test scores — and, in some cases, may actually have a negative impact on the scores of certain types of students. This study has some of the same limitations as earlier work comparing charters to district schools (including a failure to distinguish between high-needs special education students and those with fewer challenges, and the merging of students who receive free lunch vs. reduced price meals). In general, however, it is a worthwhile addition to a growing body of research which documents this pattern of uneven performance, and calls into question arguments that the charter school model should be rapidly expanded without further exploration into whether or not it truly improves student achievement.

The new research, done by Mathematica Policy Research with a grant from the federal Department of Education (a strong advocate of charters) looked at test scores and other results from 36 charter middle schools in 15 states. Using a method in which the test scores of students who won charter lotteries were compared with those who entered the lotteries but lost (in order to compare students with equally motivated parents), the researchers found that on average, charter middle schools that hold lotteries are no more successful in raising students’ achievement than the traditional public schools their students would have otherwise attended. On average, these charter schools had no significant impacts on students’ math or reading test scores, on other measures of academic progress (such as attendance or grade promotion), or on student conduct within or outside of school. However, attending a study charter school did significantly and consistently improve both students’ and parents’ satisfaction with school.

One especially strong aspect of this study is the researchers’ efforts to examine which specific practices and policies (beyond just the charter model itself) were associated with positive and negative effects on student success. After looking at a wide range of possible factors, they concluded that only a few features of charter middle schools are significantly associated with more positive impacts on achievement, including smaller enrollments and the use of ability grouping in math classes. Charter schools serving more low-income or low-achieving students also had statistically significant positive effects on math test scores (but no significant effect on these students’ reading scores), while charter schools serving more advantaged students — those with higher income and prior achievement — had significant negative effects on these students’ math test scores.

As with other research, these conclusions should be considered in the light of the methodological limitations of the report. Although the lottery-based method of studying charter school effects is one of the strongest in terms of validity, the researchers’ screening process to select schools for the study resulted in the winnowing of the total number examined to 36 schools out of the 492 which were potentially eligible. In addition, the researchers note that, on average, the charter schools in their study “served students who were more economically advantaged, less likely to come from racial/ethnic minority groups, and more high-achieving” than students in other charter middle schools. As the study does not provide the names of participating schools, it is difficult to gauge which schools their results do address.

In addition, this study also does not fully address issues of retention and attrition, which previous studies (including Mathematica’s own study of KIPP and this recent Edwize post have shown can differ widely between charter schools and district schools. The report does examine the question of peer effects (comparing the demographics and achievement of lottery winners’ and losers’ classmates and schoolmates), but concludes that the differences it finds are not statistically significant. Finally, it also fails to address differing levels of need within the categories of special education and reduced price/free lunch categories of students. For the former, the UFT’s studies of New York charters (Separate and Unequal; Special Education in Charters and District Schools) have shown that charters tend to enroll students in the lowest-need categories of special education, while district schools’ special ed enrollments tend to have higher proportions of students who need more services. Similarly, the differences in economic background between students who receive free lunch and those who receive reduced price lunch are both significant and not addressed in this study. The study’s lack of focus on these differences calls into question the relevance of some of its conclusions, especially since schools in New York City and other urban areas tend to have higher proportions of very low-income students with high special needs.

However, this study supports the argument that the charter model is not a panacea, finding (as did the earlier CREDO study) that charters vary widely in their impact on student test scores and other measures of achievement. Its finding that charters actually have a negative average effect on students with relatively high levels of previous achievement and those from relatively advantaged economic backgrounds should be especially sobering for those who have promoted the charter model as an inherently positive reform. Like the study’s authors, we agree that more research is needed to determine whether (and under what circumstances) the charter model has positive or negative effects on student achievement.