Market of opinions
The founder of the Gallup Organisation once said: "I could prove God statistically." How much should we trust these pollsters, asks Frank Jordans
Osama bin Laden knows how you feel. "Public polls show that most European people want a truce," the leader of fundamentalist Islam's most militant manifestation confidently declared in a recent taped broadcast. British media mogul Richard Desmond has similar mindreading abilities: in April his flagship newspaper, the Daily Express, trumpeted on its front page that "93% agree we have all had enough of Tony Blair".
Based on supposedly accurate opinion polls, the terrorist and the publisher announced through the medium of their choice what the public wanted and how this could be achieved: make peace with al-Qaeda and vote Tory at the next general election.
The problem with their arguments, as observant readers will have noticed, is that their figures are rather shaky. Whichever national survey you look at, none give any indication that Europeans want to call a 'truce' with al-Qaeda. As yet, there has been no panEuropean poll about the fight against terrorism.
Nor do Tony Blair's admittedly tepid popularity ratings hint that the people want his head on a stick; at the time of writing ICM Research still found that Labour would win a general election, albeit with considerably fewer seats than it did in 2001.
But such minor details do not stop anyone with an agenda from plucking figures out of thin air and claiming they prove a point. As HG Wells once wrote: "Satan delights in statistics and in quoting scripture." An educated populace is expected to know this and to navigate calmly through a sea of statistics when making up its own opinion on matters large and small. But can we trust any polls anymore? Should we filter them out as we read, even as the barrage of figures increases?
Polling in Britain has grown into a billion pound industry. Worldwide turnover now tops £20bn, and with a presidential election looming in the US that figure is sure to rise.
According to the Market Research Society (MRS), the biggest growth areas for opinion polling are social topics like politics, the environment, religion, and moral issues. In modern democracies, no government, NGO, or political party can issue policy papers any more without claiming the backing of the people.
The MRS has a strict code of conduct: researchers have to be honest with interviewers and clients, their practices have to be transparent and open to scrutiny, they must accurately acknowledge relevant sources, and people's personal data must be kept secure. Leading market research organisations also make a point of adding a short 'mustbepublished' paragraph about the sample size, date it was collected and margin of error to every news story. Some even make the entire dataset available for public scrutiny on their website or that of their client.
But beyond the good practice of the pollsters themselves, the world of statistics becomes hazier. Innumerate clients misrepresent data; figures are presented selectively; distortions become gospel. While most pollsters say they would contact clients whose analyses didn't tally with the numbers, few ever do. Only the Office of National Statistics, free of commercial concerns as it is, bothers to write letters of clarification to those who misuse their data.
Among the worst offenders are journalists: fearful of making wrong predictions over voting behaviour (as in the 1992 general election, when Labour was tipped to win but the Tories squeezed in) they now cover their backs by citing independent research, but will happily make statements about public opinion that cannot be readily verified by outcomes. A recent Guardian poll indicated plummeting popularity for Tony Blair among Muslim voters because of the war on terror: how many votes he picked up among other sections of the population was not asked.
Tabloids have even fewer qualms about statistical accuracy and political partisanship. The Daily Express poll on Blair appears to have been an inhouse job: based on nothing more than readers calling a Daily Express phoneline after reading an article rounding on Blair for his political failings. Even though we were told the phonelines were "red hot", we never learned the sample size (93 per cent of what?). Was it a coincidence that the paper had announced it was backing the Conservative Party in the next election two days before it was able to chronicle this momentous shift in public opinion?
Even if the poll is properly conducted, its interpretation can totally change its meaning. Consider the attention recently given to one carried out by Oxford Research International (ORI). Having already done audience surveys in Iraq for BBC Worldwide, the organisation was commissioned by four national broadcasters to conduct an opinion poll in which Iraqis were asked how they felt about various issues since the fall of Saddam Hussein.
Reporting the results Qatar-based broadcaster al-Jazeera headlined: "Iraqi public wants US out". For America's CNN the verdict was: "Iraqis say life better now". Meanwhile, German broadcaster ARD, which had cocommissioned the survey, declared: "Iraqis show little faith in democracy".
The survey was also cited approvingly on President Bush's official weblog as evidence that the invasion of Iraq had been justified. Opponents of the war, meanwhile, feared propaganda defeat, and so attacked not the poll's interpretation but its execution. One Guardian columnist took issue with the apparent disparity between the percentage difference of Shi'a Sunni respondents, compared to the overall makeup of the country.
According to Christoph Sahm, the director of ORI: "[The journalist] asked me why we had so few Shi'a, and I said we had a very considerable portion of people who didn't want to say which doctrine they follow and if I check their attitudinal profile against that of the Shi'a it's very similar."
Pollsters generally adjust figures to reflect socalled 'prestige bias' and weigh respondents according to the known profile of the population. In Britain, for example, Liberal Democrat voters are more likely to 'forget' they voted for the party at the last election than Labour or Tory voters. This kind of adjustment, says Sahm, is what ORI did in Iraq, thereby smoothing out the apparent imbalance and ensuring the figures were truly representative. But he adds:
"As a sociologist I would never have asked about religion, but we were forced to by the broadcasters. Now the media are beginning to dissect the sample by Shi'a versus Sunnis. They are trying to pin the current riots on the Shi'a, but if you talk to people in Baghdad it's not about the Shi'a: the Sunnis are also rioting. The rioting is not specific to the religion of the people or their doctrine. They are rioting because they are not very happy."
Sahm says Iraqis want democracy: "We are being told by some that they want religious government but that's complete rubbish. Twelve per cent want that. The rest don't, and they're very clear about it."
The survey was conducted by more than 40 local interviewers, all trained to high standard, and one in four samples was verified by supervisors to make sure no awkward questions had been left out. At 96 per cent the response rate was far higher than in western countries. "Without reservation, this was a good survey," says Sahm.
As far as the White House take on the survey goes, he is adamant: "The survey does not at all support the idea of the morality of invading Iraq a year ago." But Sahm concedes that clients and third parties can interpret the polls how they want, as long as the data is correctly presented and acknowledged.
If Iraq was the subject of the past year for pollsters, the next few months will be dominated by polls on the US presidential election. In the runup to the 2000 contest, one in three US news stories featured polls. Although most politicians say they do not swing with public opinion ("If I tried to finetune my messages based upon polls, I think I'd be pretty ineffective," Bush recently told reporters), their campaign managers are eager to highlight every percentage point lead, while downplaying less rosy results with a waitandsee smirk.
Reporters, meanwhile, turn changes in poll numbers into news hooks, headlines and even frontpage stories, something even Bob Worcester, the chairman of market researcher MORI, decries. He has described surveys commissioned purely for their headline value as a "dumbing down" of political polling.
The problem, according to Matthew Robinson, a political analyst and author of Mobocracy: How the Media's Obsession with Polling Twists the News, Alters Elections, and Undermines Democracy, is that that most people interviewed for polls don't really have any clear opinion on political issues. Recent polls in the Washington Post would seem to support his view that a simple change of wording can result in two contradictory answers.
With news organisations demanding value for money when they commission polls, researchers have been developing ever faster and efficient ways of reaching large numbers of people. Partly this is spurred on by the fact that more and more people are declining to be interviewed: refusal rates in Britain and the US now hover around 50 per cent, meaning only every second attempt yields data that can be processed, packaged and sold to clients.
Prescripted phone polling, rather than facetoface interviewing, is now the norm, and Internet surveys have taken off with the establishment of organisations such as YouGov. Dubbed 'Anything you want, Guv' by its critics, the organisation was set up by a political columnist, Peter Kellner, who had clearly seen that there was money to be made in serving the media the latest public opinions on a plate all scientifically accurate of course, despite the fact that respondents are paid to fill out surveys.
Kellner, who is the director of YouGov, departs from the usual practice of separating polling and interpretation by regularly offering his own take on opinion polls. Not only that, he dishes out advice to politicians as well. In the runup to the war in Iraq, Kellner published an open letter to Tony Blair outlining what steps the Prime Minister had to take to convince war sceptics that a) Saddam Hussein represented a clear and present danger to Britain, b) the antiwar crowd constituted a minority of Britons, and c) that Blair was not America's poodle, but had the backing of the world community. Pollster, pundit, and government adviser all in one.
Faced with such conflicts of interest, the fact that every poll invariably receives a political spin from those who have paid for it, and PR companies who, according to one market researcher, openly ask for favourable figures to back up their products, is there any reason at all to go on taking polls seriously?
Christoph Sahm believes good polls are still crucial: "In Iraq we now have the huge advantage of the perspective of 6,000 households across two surveys; a TV camera can never have that perspective. As far as I am concerned, social research and journalism need to work together, but that necessitates the results being reported honestly."
Meanwhile, the director of the Economic and Social Research Council, Ian Diamond, is blunt: "We have to know where the data comes from. We have to know the basis on which it is collected. We need to stop being an innumerate country, and start to get on board the fact that it is perfectly possible for everybody to understand the basics of statistics. The sooner we start doing that, the sooner we will be able to challenge and question data properly, use evidence effectively, and not say: 'Oh my god it's a number, I don't know what it means'."