The Mismeasure of Bias

This is one of the most astute critiques of a methodologically flawed paper that purports to measure ideological bias in the media. Nyhan's main points are:

(1) Technocratic centrist to liberal organizations like Brookings and the Center on Budget and Policy Priorities tend to have more credentialed experts with peer-reviewed publications than their conservative counterparts. This may result in a greater number of citations by the press, which seeks out expert perspectives on the news, but not more citations by members of Congress, who generally seek out views that reinforce their own.

(2) The Groseclose/Milyo methodology doesn't allow for differential rates of productivity in producing work of interest to the media or Congress between organizations. To the extent that a think tank is better at marketing itself to the press than Congress (or vice versa), it could skew the results.

He quotes other sources, too, and this argument from Media Matters is particularly convincing:

For instance, according to their data, the National Association for the Advancement of Colored People (NAACP) is the third most-quoted group on the list. But stories about race relations that include a quote from an NAACP representative are unlikely to be "balanced" with quotes from another group on their list. Their quotes will often be balanced by quotes from an individual, depending on the nature of the story; however, because there are no pro-racism groups of any legitimacy (or on Groseclose and Milyo's list), such stories will be coded as having a "liberal bias."

And there are more general issues that would remain even if these details were somehow fixed. The very notion that some average (median or whatever) of population's views is "unbiased" is unfounded; and, even if the population's median view were deemed "unbiased" by definition, there would still be a big leap from the median member of Congress to the median voter, and another from the median voter to the median member of the population.

Too bad Andrew Gelman's endorsement of Brendan Nyhan's critique is only lukewarm, even though he realizes that "bias" is a misnomer for the quantity that G&M measured. More generally, it is sad that so often people with great analytical skills fail to appreciate the world beyond the limitations of a mathematical model.