Paglia and standardized tests

In the first part of this series, we're going to look at evidence from scores on standardized tests to question whether or not reading skills have been going down. This is an issue which is wholly separable from the question of what might cause it, so in making the arguments to follow, the Cartoon Network is not to be assumed guilty of anything.

As Prof. Liberman correctly points out, verbal SAT scores have been trending down for a long time. On page 7 of this study by the College Board, it can be seen that, over the period from 1967 to 2002, the mean SAT scores for all test-takers have gone from 543 to 504 on the verbal section, while the math section has taken a U-shaped trip that begins and ends at 516. This takes into account the renormalization of scores that was done in 1995, which would have given SC about a 1530 in the current scoring regime. Alas, he took it in 1993.

Radagast cautions (in offline discussion) that this may reflect the fact that more people are taking the SAT now than did in 1960, and so the decline may reflect a less pre-screened pool of takers, where the fact of not going to college kept a lot of people who might have taken the test from doing so. The report indeed notes that 2002 was a record year for the percentage of high school graduates who took the SAT. However, the study also finds that verbal scores flatlined in the '90s, while math scores were busy recovering. Paglia's not arguing that people have become stupider, just less literate. She even credits "the extraordinary technological aptitude of the young", which we might assume correlates better with math scores than verbal ones. SC hopes she doesn't just mean that the guy that her school's IT department sends to fix her computer is younger than her. Regardless of any claims about technological skills, though, if it was really the case that the test pool was getting weaker overall, and not just verbally, we wouldn't expect to see a recovery in either subject -- just a march straight down towards a new plateau.

The SAT trends are also reflected in GRE trends, and it's a lot harder to argue that the GRE pool is being diluted by marginal students who aren't really college-caliber. We'll start with a study commissioned by a group called the Commission on Professionals in Science and Technology. As can be seen in Table 1 of this study by Zumeta and Raveling, in the period from 1989 to 1998 -- after the SAT verbal scores had stopped falling -- the aggregate mean of GRE verbal scores among students entering all fields dropped by 26 points, while the verbal and analytical scores remained essentially unchanged. As the authors point out, the GRE standard deviation unit is 100 points, so a 26 point drop is not enormous -- but it's not just noise, either. While your host couldn't find an official College Board statement of the SAT's standard deviation, table 3 of this paper published by them suggests that the SAT's standard deviation is also about 100 points. Zumeta and Raveling's work controls for U.S. citizenship, so the GRE drop can't be attributed to an influx of non-native English speakers. Note that GRE data from before 1981 can't be reliably compared to later GRE data, because of significant changes to the format and the reporting of scores. They're not wholly incommensurable, but nobody's paying SC the going rate required for him to figure out how to massage the data for these purposes. Suffice it to say that the GRE trend confirms the SAT trend.

The Association of Neuroscience Departments and Programs did a study in 1998 of average GRE scores among applicants to neuroscience programs -- presumably a smart group of people -- and found that, while aggregate GRE scores were roughly unchanged between 1986 and 1998, the quantitative portion of the score trended upward over that time, while the verbal score rose a bit, and then declined to be noticeably lower at the endpoint of the study than the beginning (see table 2b of the linked page). For reference, the aggregate scores in 1986 were 1838; in 1991, 1865; in 1998, 1885. Interestingly, the analytical section of the scores trended upward along with the quantitative one; it's the sum of the increase in those two science-oriented sections of the test that accounts for the rise in GRE scores. Unlike the Zumeta and Raveling study, the neuroscience results aren't controlled for citizenship; we might expect that an influx of non-native English speakers would lower the aggregate verbal score. In fact, non-native English speakers' verbal performance is terrible compared to native speakers': table 10 of this 1996 sampling shows that in the verbal section of the GRE, the mean for international students was anywhere from 29 points lower (for business majors) to 102 points lower (for arts and humanities majors). This phenomenon is very much restricted to the verbal section; table 11 shows that international students across all undergraduate majors absolutely crush U.S. citizens in the math section. While international students score lower in the analytic section (table 12), it's not by too much, and the results are arguably confounded by the English skills needed to correctly interpret the questions in that section. So it might seem that non-native speakers are dragging down the neuroscience verbal results -- but here comes the gotcha:

As page 42 of that same 1996 study indicates, non-native verbal scores spent the 1982-1995 period rocketing upwards, by an increase in the mean of some 59 points among students in the biological sciences. In fact, non-native GRE verbal scores rose for all international students over that period, even while the U.S. citizen-only scores were stabilizing or dropping. Over this same period of time, the proportion of international students taking the GRE also rose, from 17% in 1982, to 24% in 1996. It seems likely, then, that the Zumeta and Raveling study actually underestimates the decline in the verbal abilities of U.S. citizens going on to graduate school -- admittedly, the years don't all line up exactly in this group of studies, but if noncitizen performance is steadily improving, and they're making up a greater proportion of the graduate population, and yet verbal performance is flatlining or falling, then U.S. citizens -- presumably native English speakers -- are getting worse. One might object that even though the internationals are improving, they're doing it from a lower starting point, and that's fair. But they're still making an enormous contribution towards raising the mean, which is declining despite the gains among the worst test takers. And for those of you speculating that maybe SC isn't accounting for students from Canada and the U.K., check out page 5 of that study -- less than 7% of all international GRE-takers came from those countries. The contribution from Australia doesn't get that figure above 8%, either. International English-speaking students are not distorting the figures; the international cohort really is gaining on the U.S. one, and yet the aggregate statistics for GRE scores are falling.

Unfortunately, stats like those above for the neuroscience community are hard to come by for other fields; while individual schools publish lots of data on the web about their own students, finding aggregate data from a community is more challenging. It's possible that digging through back issues of Psychometrika would help expand this analysis to additional fields; however, as we've already seen, the question isn't "are American students verbal scores declining?", because the answer to that is already "yes". The question is "is it even worse than we think?".

The final objection that might be raised is that, while the decline in SAT and GRE verbal scores is noticeable, it's not yet so significant as to have dropped off by a whole standard deviation. It may be a bit of a weak peg to hang the argument on, but I think the persistence of the data is a lot more meaningful than the exact magnitude of the change. The contrast in the math and verbal trends of the SAT suggests that Paglia is right about what's happened to verbal skills in our society, but that her analysis of why happens to be wrong. It might be argued that the '70s drop in math scores had to do with the disastrous "new math" pedagogy, and that a return to more traditional mathematical education brought high-school skills back to where they were. At least the math scores bounced back. Even though they bottomed at around the same time, there has been no such rebound in verbal scores, and the fact that it hasn't happened in 20 years of gradual recovery of math skills suggests that we're not just getting stupider on the whole. That doesn't bode well for Camille Paglia's claim that TV and video games (insert angelic chorus of "ah!", just like at the start of The Simpsons, here -- I'm sure Ms. Paglia would appreciate the irony of that reference) are responsible for ruining the brains of young people. But as you can read here tomorrow morning, there's some startling evidence that she's right about a "massive transformation in Western culture", and that it's likely to continue to result in poorer verbal skills.

Comments

[sahara]

Oh lord!

If the "massive transformation in Western culture", is likely to continue to result in poorer verbal skills––it may be that young people simply aren't talking to each other in a way that requires a broad vocabulary.

In the push to "market" products to such a large spending group, advertisers have been speaking slang and ebonics to kids since birth. What do folks expect?

I'm from the minority generation that had "two languages"––the language I spoke with my people, called "black english" at the time. And the language I spoke with the broader "white" world. And, like, ya'know? was out of the question. Vocabulary was looked upon like painting with a variety of colors. Why be sad when you could be chagrined, pensive, despondent, etc. You get the picture.

At present, most of my co-workers at the fashion mag I work at, are 35 and under. They've confided to me that my verbal skills intimidate them, as they grew up speaking TV, chat, and now, marketing. Pop culture has now become an art in itself, no longer relegated to the lower classes.

When I watch and played video games with my cousins, I found that they use a different kind of intelligence––more suited to be soldiers in the military, in the future, I guess. But they are definitely NOT stupid.

Maybe Camille should play some. Some adults are just mad, that their kids aren't talking to THEM.

[Dustin]

The night of the fight, you may feel a slight sting. That's pride f*cking with you. F*ck pride. Pride only hurts, it never helps.
121deed8c474aefa04cadb8b4a327d70

[sildenafil citrate]

take the right measures is the best way of action in this cases.

[GRE Prep]

Thanks for sharing this article here, this is very useful especially that I am on the same scenario right now.

[Mordecai]

When figuring out the "contribution towards raising the mean" of international students, you seem to have overlooked something. The rate of change of the mean should be the sum of three terms: (1) the rate of change of the US mean, times the fraction of test-takers from the US; (2) the rate of change of the international mean, times one minus this fraction; (3) the rate of change of the fraction, times the difference in the two means. If the proportion of test-takers from the US is dropping sufficiently quickly, and you claim that it's indeed dropping, a negative third term can potentially swamp the positive second term.

An example: say the US mean is 1 in 1990 and 1 in 2000; say the international mean rockets up from 0.7 to 0.8, and during the same period the proportion of test-takers from the US drops from 0.9 to 0.8. Then the mean goes from 0.97 to 0.96, despite no group getting worse.

Not to say this is happening, but recall your argument was "it's worse than it looks." Instead, it could reasonably be better.

[Build Your Credit]

for those of you speculating that maybe SC isn't accounting for students from Canada and the U.K., check out page 5 of that study -- less than 7% of all international GRE-takers came from those countries...

[yampools]

This phenomenon is very limited which shows that international students in all degree majors are absolutely crushes the citizens of the United States in the math section.