Econometricians are often pretty smart at thinking up ways to measure things. I recently attended a seminar by Professor Matthew Gentzkow from University of Chicago Graduate School of Business who is doing research on the vexed issue of media slant. You might think that media slant or media bias is completely in the eye of the beholder. There is no âfair mediaâ to anchor against I hear you cry. But economists have no need of such notions as fairness. They look for comparisons, differences. And their anchor is âwhat would a rational utility maximiser do?â The bottom line ends up being : âIf it looks like a goat and sounds like a goat, its probably a goatâ.
For those of you who have limited time to read the paper, my executive summary is below.
First, you can measure newspaper bias in a natural and objective manner – natural because the final measure has a simple and natural interpretation, objective because it is done by a computer not a person. The idea is to build a text processing algorithm whose aim it to to classify members of congress as republican or democrat on the basis of their use of certain phrases in their recorded utterances in the house. For instance, what republicans might call the War on Terror, democrats will call the War in Iraq. You then apply the same algorithm to the news text* of newspapers. And you end up with a measure such as âif the SMH were a member of parliament it would be 71% likely to be an ALP member.â Gentzkow and Shapiro have done this for a wide range of US papers. Pretty damned clever, heh?
Second, are papers across the nation collectively biased? Both left and right have argued that they are (in opposite directions of course). Well, it turns out that collectively they are slightly left leaning, in the sense that the US newspapers collectively have an average slant that corresponds to a 47% republican complexion whereas during the period of evaluation republican support was around 53%.
Third, is this bias politically driven, for instance by the reporters? Economists would like to explain actions in terms of utility maximization. And newspapers are for-profit businesses so their utility has a lot to do with profit which means sales. If we start with the idea that left/right readers like to read left/right papers and if the left are better educated and read more, then the slight left leaning bias could just be a rational market response by the newspapers. The authors look at the relationship between slant of papers and the political leanings of their geographical catchment area and find that there is a clear relationship. Further, they estimate that if newspapers were rationally maximizing profit within each catchment then the average political slant should turn out to be equivalent to a 46% republican complexion. And the papers are slightly to the right of this profit maximizing point.
Finally, do owners of papers impose slant? Statistically, we would ask whether the variation in slant from newspaper to newspaper can be explained by ownership. It appears not from the data. You do not get a better predictor of slant by including ownership in the regression, and there does not seem to be significant correlation within stables of papers with a common owner, after other things like geography are taken into account.
I would love to have the time to do a similar study in Australia. My perception is that the Oz is blatantly biased on the issue of workchoices and the SMH is blatantly biased on any issue around social progressiveness. But the bias is just as likely to be within my own skull.
* The algorithm is only run on news text, not opinion pieces or editorials. One may argue that bias in these is less important since it is overt.
If Bush were to use the phrase “War on Terror”, or something clear as that textually, at every opportunity, the phrase would be bleached of its impact. (It’s diminished, already). Surely what happens is a (textually) definitive phrase is injected, and through the immediate while is then rounded out. Can an algorithm fully embrace all of that?
Very interesting – thx Chris. A lot of the bias in the Oz is via comment. Simply allowing outrageous comment – like Janet A’s recent effort has some effect I think in shifting the barrier between the sayable and the unsayable – the respectable and the unrespectable.
Robert,
I think the beauty of the approach is that it can embrace a lot of subtle verbal strategies without having to identify it in advance. In asking the simple question “does this paper sound like a republican or a democrat” we focus attention on the germance issue. Republicans and democrats might change their verbal strategies over time and the algorithm would accomodate that. So long as we parse the text with enough care we should be able to pull out the main differences.
Of course, there are subtle differences that simple minded text analysis will not pick up. Text is always much more than a sequence of words. But simple texdt strings may be effective surrogates for a whole lot of meaning. For instance, if we were parsing blog comments I reckon that the text strings “rwdb” and “teh left” would pretty effectively classify the commenter. Applying this principle to my own comment I would be classified as unclassifiable!
Fascinating, Chris, thanks. How would this go assessing John Howard’s text? His words translate with awesome power to the page: quotes from Howard are incredibly consumable, yet they don’t always state a bias. He presents a case, as “facts”, and these often merely suggest a cause – the power of suggestion – while maintaining the appearance of being middle of the road, balanced, etc. That is, his tremendous communicative ability over the years has been to create in peoples’ minds through the power of suggestion a certain slanted take-out not evident in the text of what he says.
Which means that if a paper defaults to repeating his words, without challenging them, a bias is created which cannot be objectively measured.
Opposition necessarily must challenge what the incumbent says, which I would imagine (do you agree?) would necessarily slant the text reading to a “Left” bias.
Would you see a challenge here, or is this sort of thing covered?
I find it hard to sustain such conclusions with such data. Suppose the world works like the following:
– a fringe group creates trouble
– a right-wing allience of military industrialists, social conservatives, religious fanatics, and people with a chip on their shoulder decide to go nuts. They call everything connected with the fringe group, however distant, terrorism, and declare war on it.
– the allience, with the help of the media, wins. The whole country is fighting this war and waving the flag. However, the left-wing, simply to signal it is more reasonable than the allience, distinguishes itself by refusing to use the word terrorism in every sentence and instead moderates itself to using it every other sentence. They still use the word war in every sentence though.
– an econometrician with just enough nouse to get a search engine on some words going ‘discovers’ that the media uses the words of the left-wing slightly more often than the rigth-wing. He convinces a journal editor to publish his conclusions.
Ergo a left-wing bias? Whilst perhaps up to 95% of the lingo used comes initially from the right-wing, one concludes from the division between the remaining 5% where people still use different terms that there is a left-wing bias? Even if the world is only a bit like I sketched, I hope you see the stretch between the data actually used and the conclusions drawn. To use an old phrase used recently by Jacques Chester ‘every advanced new technology seems like magic’. Dont be fooled too easily….
I understand your point Paul/Robert. You are trying to anchor bias on an absolute and pointing out that comparisons of text will not pick up such bias. An even better example might be that a text parsing algorithm would not pick up bias in the Austrian papers of the 1930s. If the body politic becomes deranged I think we are collectively in trouble.
What the authors provide is a way of objectively comparing newspaper discourse with open discourse in parliament. If parliament represents some kind of free political equilibrium then it becomes a natural anchor. Deviations from this become slant. What else do we mean by slant?
I think you may be arguing that in some situations, such as Austria in the 1930s, media slant – defined as deviation from the current political balance – would be a good thing. I would not disagree. In fact, I largely agree with your comment and am defending the methodology for academic interest.
I think that Nick’s comment is more of a problem for this methodology. He is referring firstly to opinion pieces (which were excluded from the analysis) and secondly to the damaging meta effects some public commentary can have. Such effects, such as those of derogatory racsist language, cannot be captured by a static text analysis.
Paul, what is “current political balance”? The reason I ask is that, as an example, in the domestic situation over the last decade, Howard’s government has edged us incrementally through some drastic changes. Not harping on Howard here for the sake of it, but if we’re talking language and bias, Howard is an excellent port of call to explore. I’d contend again that JH has not exhibited slant in words to anywhere near the extent of his actions. These are words spoken as broadcast, which translate as mentioned above, and not counting parliamentary speech (of which I’m uncertain in this context). So taking it merely as broadcast speech translated to press, the current political balance is continually changing in deeds, but not in words. If you’re talking about an ideological slant, again, Howard’s words don’t speak of an ideology as much as they speak blandly of a middle ground to be readily consumed by his target market.
To me, on media face value, there is a disparity in measuring “current political balance” and the words used, through time. This means the definition of ‘slant’ rings as unfounded.
This would indicate that you are measuring successfully as, say, on and within a boat, but the boat moving along a river in a wider context is not being taken into account.
Perhaps I am missing something here?
And are you saying that the words used in Parliament form what is effectively a control, by which all other text except in opinion pieces can measured?
Apologies, Paul, that’s for Chris. Was caught up in the concept.
Chris, the assumption above is that JH speaks in parliament in ways which are concerned for public consumption: that he has an eye for the market while he frames his words. Not knowing the complete manner in which Howard speaks in parliament, do you know this, and would you be confident the algorithm as created by parliamentary language still usefully applies?
First, as pointed out by commenters above this technique only measures the relative position of a newspaper by comparison to the dominant political parties; it tells us nothing about what the positions actually mean. Isn’t this the kind of moral relativism we are all supposed to deplore these days?
Second, the reason broadsheets run a soft-liberal line is indeed purely commercial. That’s where the AB demographic readers are. The inability of some on the right to understand this basic application of the law of supply and demand is puzzling (where it’s real and not just confected outrage).
There is some futher discussion of some of the practicalities of applying this idea in an Australian context at http://dsanalytics.com/dsblog/can-text-mining-detect-media-bias-how-hard-can-it-be_98.