A FALLACY OF REGRESSION TO THE MEAN is nicely illustrated, I think by something in the Chronicle of Higher Education's May 16 daily briefing (yes, I'm behind):
The SAT is not making the grade, says Roy O. Freedle, a former senior research psychologist at Educational Testing Service. He writes that the test is biased against minority students and needs to be reformed to more accurately represent their achievement and potential. Mr. Freedle compared the performances of black students and white students on what are considered the easy questions and hard questions on the test. Among students who had received the same overall score, he says, the black students had consistently scored a little better on the hard questions and a little worse on the easy ones.Here's a stylized story to illustrate the fallacy. Suppose we have two kinds of questions, 50 easy and 500 hard, and two kinds of students, black and white. Suppose, too, that the black students don't know the answers to any of the questions and that white students know the answers to all the easy ones, but none of the hard ones. If a student doesn't know the answer, he guesses, and has a 1 in 5 probability of getting the answer right anyway. This means that the average white score will be 50+100=150 and the average black score will be 10+100=110.
Suppose a black student and white student each score 120 correct. The white student has gotten 50 easy questions correct, and 70 hard questions. His guesses were unlucky. The black student has mixture of types of questions correct-- maybe, 10 easy and 110 hard. His guesses were luckier. The white student does better on the easy questions, because of his higher ability, and worse on the hard questions, because of his worse luck.
Does this depend on whites being unluckier than blacks? Not on average, because what choosing students with the same scores does is to select for especially unlucky white students and especially lucky black students. For a white student, 120 is a below- average score, so white students with that score are the unlucky ones. For a black student, 120 is an above-average score, so black students with that score are the lucky ones.
This is a stylized version of what is going on, of course, because whites differ and blacks differ, and questions are on a continuum from easy to hard, but you should be able to see the underlying principle.
The article is: Roy Freedle, "Correcting the SAT's Ethnic and Social- Class Bias: A Method for Reestimating SAT Scores," Harvard Educational Review, Spring 2003, Vol. 73 Issue 1, p1, 43p, 9 charts.
[ http://php.indiana.edu/~erasmuse/w/03.05.26a.htm ]