Polling Accuracy - Which firm did best?
July 21st, 2004
-
Populus sets out its case
Opinion polls are critical tools for political gamblers and in recent weeks we’ve made a number of observations about their accuracy. In particular we have looked at what happened at the Euro Elections last month when surveys by YouGov, in the Telegraph, and Populus, in the Times, could be compared with real results.
-
Two days before the Euro vote we noted that YouGov was putting its reputation on the line with its very high figure for UKIP and, indeed, this is what is has been attacked for.
But after the elections we also noted that the internet pollster got closest with the crucial Lab-Con split although both it and Populus both under-estimated the Tories and over-estimated Labour - something that most polls most of the time when tested seem to do. This tendency is something that gamblers should always be aware of and should always factor in.
This prompted the head of Populus, Andrew Cooper, to post this comment. We think that he makes important points that should be given greater prominence.
As a matter of actual fact the accepted measure of accuracy in the polling industry is the average error in the final poll published before an election. On this measure the Populus poll conducted nearly a week before polling day, was fractionally more accurate than YouGov’s poll conducted around the same time, with an average error of 2% compared with 2.2% for YouGov’s.
What is more YouGov’s exit poll - with a massive 7,000 sample of people supposedly being polled after they had voted, and therefore in theory much more accurate, had an average error of 2.8% (compared with a theoretical margin of error, given sample size, of 1.1% - which technically classifies it as a rogue poll). The YouGov exit poll predicted essentially a 3-way tie between Labour,Tories and UKIP.
Within all of these figures lies an immutable fact about polls, which you seem to have missed: they are blunt instruments; even the most methodologically robust poll cannot be more accurate than its standard deviation, which is a function of sampling. For most polls this means the figure for any party is accurate to within plus or minus 3%. So a poll putting Labour on, say, 32%, in fact means simply that there is a 95% probability that their support lies somewhere between 29% and 35%.
For this reason reading the leads between parties from polls is an even more treacherous business, since the margins of error on two figures are compounded: our last poll, for example, showed a Labour lead of 4% but in fact, when margins of error are taken into account, it meant that Labour’s lead is (at 95% confidence) somewhere between -2% and +10%.
Generally speaking the polling companies have done a pretty good job in as much as they’ve stayed within the margin of error most of the time. Populus did so again in the Euro elections; YouGov didn’t. Not only was its average error outside the defined margin of error, but it erred on its UKIP rating by a wide margin - an egregious mistake (presumably a result of the sampling problems associated with internet-only polling), compounded by the probability that UKIP would not have done even as well as they did by the end had not
YouGov not wrongly placed them at nearly 20% right at the beginning, which became the story of the campaign. Polls which make headlines, as that one did, have the capacity to become to a degree self-fulfilling: just one of the reasons why integrity of method is so important and why there is a drive now to improve the self-regulatory standards of the polling industry.
Incidentally, since you raise the point, Populus has only once shown a Tory lead - in May. ICM have shown the 2 main parties level-pegging several times. MORI have shown a Tory lead at least once (July 2003). And if you average each company’s polls (a sensible way to allow for the inevitable blips) you will find that, since last summer Populus has shown an average Labour lead of 2.9% compared with 3.9% for ICM, and 2.9% for MORI while YouGov has the Tories in the lead by an average of 2.4%.
The key driver of this variation is the fact that Populus, ICM and MORI all have the Tories averaging 32% over the last 18 months, while YouGov have them averaging 38%.
Your proclamation that Labour will win most seats at the next election but might not have an overall majority puts you way out on that limb with YouGov! All I can say is I’m very glad I don’t have any of my money riding on your prediction (or their’s)
We are grateful to Andrew for this but he has not touched on the major issue - why is it that almost without exception polls are biased to Labour? We have made critical comments in the past about other firms and none has responded. This is an important issue and the more we know the better.
-
We hope that at least one of the bookmakers will open a market at the General Election on which pollster will get closest
.
Image http://www.aperfectworld.org/clipart/Metaphors/signonlineb.gif
MessageSpace Advertising
1 - Andrew Cooper\’s point about the only fair comparison being the last full poll before the election is reasonable (even allowing for a certain vested interest in this case). From a gambling point of view it would have been helpful to have further polls closer to polling day.
2 - Exit polls are of use when the betting market is to stay open for a long time between the close of polls and the declaration of the actual results (as in the Euros). In General
Elections however the exit polls are superseded by real results within a couple of hours.
3 - As for the margin of accuracy, I have a question. Say a party is given 32% in a poll with a 3-point margin of error. Andrew Cooper explains that that means there is a 95% chance of their real support lying between 29%-35%. Fair enough, but, mathematically-speaking, can one infer that 32% is the most likely figure? Is there a bell(?) graph starting at 29, peaking at 32, and falling back down to 35? Or are 29%, 30%, 31%, 32%, 33%, 34% and 35% all equally likely to be the true figure?
In fact, Yougov have proved remarkably accurate in their prediction of the Conservative vote share in elections throughout this Parliament, not just in the Euros, but also in the London local elections of 2002, and in the Scottish Parliamentary elections of 2003.
On Villan\’s third point, my understanding is that it is indeed a bell curve. This certainly makes sense intuitively. 95% confidence that the true figure lies within a six point range means a huge tail on either side in which there is only a one in twenty chance (in total) of the true figure falling. This tail goes all the way down to the extreme case that your 1000 person poll picked on the only 320 people in the country who intend to vote Labour. This is obviously incredibly unlikely, but as you move up the tail and the implied overestimate of Labour support becomes less extreme, it makes intuitive sense that there is progressively more chance that the figure you are thinking of is the true figure (and likewise back down the tail on the other side of 32%).
Yes, thank you, \”bell-curve\”! That was the term I was trying to drum up.
With seven possible results within the range 29%-35%, the chances of the headline poll percentage actually being CORRECT cant be more than about 20% (the left-hand side of the graph might read, say, 5, 10, 15, 20, 15, 10, 5).
In other words, a \”Tories on 32%\” headline actually means \”We calculate there is a (roughly)20% chance the Tories are on 32%\”.
Apologies to the pollsters if my reasoning is wrong - although actually the problem is more with the way the polls are reported than with the polls themselves.
For what it\’s worth, YouGov have also published figures claiming to show the relevant margins of error. There is something of a contrast.YouGov put their own margin of error at 2.3%, slightly higher than Andrew Cooper\’s calculation although pretty much the same. In contrast they have a much higher figure for Populus\’s margin of error. I wonder why?
It\’s a rhetorical question actually because I know the answer. Assuming Andrew is referring to the Populus poll carried out between June 4th and June 6th, the headline figures published in the Times did not provide figures for Labour or the Liberal Democrats, instead they provided ranges (25-26% and 16-18% respectively). If you take the lower end of these two ranges you get a margin of error of 1.97% - rounded up to 2%. On the other hand, if you take the mid-point of the two ranges (Labour 25.5% and the Lib Dems on 17%, you get a margin of error of 2.7%, significantly less accurate then YouGov\’s effort. (YouGov also round up the Labour figure to 26%, making Populus even less accurate. That is pushing it a bit to be honest).
Suffice to say, depending upon which figures you use to calculate the margin of error - you get different pollsters coming out top.
LOL - thanks for that.