I recently read a column by a syndicated columnist named Mona Charen that was so audacious and so clever in its misuse of statistics that I almost feel compelled to create an award in her honor and make her the first recipient! The column, which was her first of 2004, was titled Education follies (http://www.townhall.com/columnists/monacharen/mc20040102.shtml.) The column endorses the use of school vouchers. To support her view, Ms. Charen referenced a “randomized study examining the effect of school vouchers on African American youngsters in New York.” The study was carried out by Paul E. Peterson of Harvard. Let me be clear that my concern here is not about school vouchers, politics, philosophy, or any other subjective aspects of the column. I am also not concerned about Paul Peterson or his study. I assume that as a professor at Harvard he does not make a habbit of abusing statistics to support his views if for no other reason than his colleagues would skewer him! My soul interest is in Ms. Charen’s description of the study and its results.
Before quoting from Ms. Charen I ask the reader to consider how he or she would construct a statistical study to measure the effects of school vouchers on African American youngsters in New York. A prototype for such a test might be constructed along the following lines. Two “sufficiently large” disjoint groups of African American youngsters from New York are randomly selected. “Sufficiently large” means that the possibility of significant statistical differences in the makeup of the groups can be ignored. Vouchers would be offered to one of the groups and not offered to the other. The students would be tracked as they progressed through their schooling over some period of time. The children in each of the groups would then be given a common test and the distributions of scores for the two groups would be compared. The length of time before testing, the nature of the test, and the manner of comparing the distributions would have to be decided. Of course this level of experiment couldn’t account for issues like whether the approach would scale to large numbers of students but at least it would give us some preliminary indication of the viability of the idea.
Here’s Ms. Charen’s description of the study and its results:
The study began in 1997, when the School Choice
Scholarships Foundation offered vouchers to 1,200 New York City public school
students in kindergarten through fourth grade. The scholarships were worth
$1,400 annually. The students who received the vouchers were similar in every
way to those who did not. Eighty percent came from single-parent families. The
results were impressive. Students who received vouchers scored one grade level
higher in reading and math than students in public school.
Do you see what she’s done? Is it not clever? If you didn’t see it I can understand. That’s part of the beauty of this lie. The key is the switch from offered to received. The comparison was made not with the entire group of students offered the vouchers but only with those who received the offer. Do you suppose that might have introduced just a tad bit of bias into the sample sets? Let’s see, the families of the students who received the vouchers expressed a sufficient degree of commitment to the education of their children to supply the additional thousands of dollars required beyond the tuition covered by the stipend. Do you suppose that level of commitment to education by the parents might in itself have been a factor in the success of these children? Do you suppose children of such parents (single or otherwise) would have scored significantly better statistically than the class as a whole whether or not they had stayed in public school? It certainly seems likely.
I have no idea how Ms. Charen came up with this thoroughly misleading description of the results. I am confident that it didn’t come from a Harvard professor. I assume that it must have been her misguided attempt to interpret the study? Ms. Charen is a nationally syndicated columnist who once worked in the white house. Surely she could have called Professor Peterson and gotten a proper description of the results or at least run her description by him to ensure that it contained at least a shred of content. I conclude that either she was purposely deceitful or just careless. Even the latter is inexcusable in my view.
Whether intentionally or not she has seriously poluted the discussion of the crucially important topic of how best to educate our children with a lie.