Guest Slot - Rod Crosby on the Kalman Filter
December 29th, 2006-
Wouldn’t it be nice…..If we could get from here…..
-
…to here?
Well, perhaps we can…. I just DID!! The second graph was created using only the few datapoints from the first graph, and a clever box-of-tricks called the Kalman Filter.
What on earth is a Kalman Filter?
Believe it or not, it is a statistical routine that originated in the 1960s in the fields of engineering and signal processing. The Kalman Filter is mathematically proven to be the optimal way of combining noisy data, and is used in many real-word applications - for example, radar and the inertial guidance of rockets. It was a Kalman Filter that took the Apollo spacecraft to the Moon and back.
In 1999, two Yale politics professors, Donald Green and Alan Gerber, hit upon the idea of using the filter to remove “noise” from opinion polls.
They have very kindly made their number-cruncher available online for anyone to use – it’s called SampleMiser
In February 2006, Gallup began including Samplemiser results in its Presidential Approval rating, and in November amateur pollsters claimed success with the technique to precisely forecast the US Senate race.
Samplemiser does an outstanding job not only in giving the best estimate of true support, but in also identifying with amazing precision those points in time where opinion turns…
So I thought I’d put it to the test using basic poll-data for the Tories. The really clever thing Samplemiser did was to take 110 irregularly-spaced polls since May 2005, and create 580 new polls - apparently out of thin air – one for every single day. In other words it enables day-to-day tracking of Tory support…
What tale does the Samplemiser tell?
Out of a blizzard of dots, on 29th September 2005, Samplemiser identified a reversal and sharp upward trend in Tory fortunes. On that day, the TV news was plastered with images of Walter Wolfgang being manhandled out of the Labour Conference, giving the party appalling publicity as perceived “bullies.”
By November the Tories were faltering, but a turn in their favour occurred when the media identified David Cameron as favourite to be the next leader. With uncanny accuracy, on the very day Cameron was elected, 6th December 2005 – Samplemiser projected a 1.5% leap in support, and the Tories soon rose like a rocket, putting on 4 percentage points in a month.
By the first week of 2006 though, there were signs of a levelling-off and slight fall-back at 37%. But the sudden defenestration of Charles Kennedy on 7th January gave Cameron a further boost, with the Tories cresting at 39% on 3rd February. However, this was the same day the leaderless LibDems pulled-off a sensation at Dunfermline, and by the very next day Samplemiser had the Tories falling sharply…
Cameron’s renewed espousal of Green issues during February appeared to pay dividends, but Samplemiser noted the abrupt end of this recovery on 4th March, with the election of Ming Campbell as LibDem leader. The Tories achieved another brief upward spike on 17th March, after 75 Labour MPs rebelled on the Education Bill.
On the 18th April 2006, the “Dave the Chameleon” Political Broadcast was aired for the first time. It proved a highly potent message, as the Tories went into free-fall on the 19th, dropping almost 3 points in 3 days, according to Samplemiser….
If our little timeline had ended there, David Cameron might be just a historical footnote by now. His short honeymoon was certainly over, the LibDems had miraculously revived, and the Tories now stood at below 33% - less than they had achieved at the General Election. Labour had appeared to land a knock-out punch with its scathing party political broadcast, and another bout of Tory in-fighting looked inevitable.
FATE, unbeknown however, was riding to the Tories’ rescue – in the ungainly, ludicrous form of John Prescott….
News of the his affair with Tracy Temple broke on 26th April, and by the 28th Samplemiser had the Tories surging by 1.5% - the start of one of the most rapid political sea-changes in recent history. Over the next 5 weeks the Tories would put on nearly 7 percentage points, peaking at just over 39% by the end of May. Further spikes in June and July were recorded, apparently related to the Prescott/Anschutz story, but interestingly the Tories seemed unable to penetrate a glass-ceiling at just over 39%.
There were fewer polls than usual in July and August. The Levy arrest didn’t register, as it fell inside an unusual poll-free fortnight, but Cameron’s “hug-a-hoodie” speech may have caused a dip in Tory support. The terror alert in mid-August could have actually boosted the Tories, but the apparent breaking of the plot propelled the Conservatives on to the downward path from the 24th August, the day after the suspects appeared in court.
The next sharp upturn in Conservative support identified by Samplemiser occurred on 7th September, the very day Tony Blair finally announced his forthcoming departure. This boost proved short-lived – less than a week - and the Tories resumed a steady decline.
On 20th October, Samplemiser recorded another surge in Tory support – in the week that Blunkett’s sensational memoirs were published, and Sir Richard Dannatt made his outspoken criticisms on Iraq. However, on 3rd November it was reported that David Cameron had returned to the “hoodie” theme. His new comments were widely ridiculed as “Love a lout.” With seeming laser-beam accuracy, Samplemiser showed Tory support plummeting, falling by over 2% from 5th November.
Since then, the Tories have hovered around the 36% mark. We have had a remarkable view of the rollercoaster ride in Tory fortunes that the poll snowstorm conceals, and only time will tell in which direction they head off next.
But one thing seems clear – to keep firm track of them we need the help of the amazing Samplemiser….
Rod Crosby has been a Lib Dem activist but says “I have never voted Labour in my life, but if by some chance there was a danger of David Cameron becoming Prime Minister, I would not hesitate to do so.“
MessageSpace Advertising
Rod - thanks for this which is very interesting. I have two points - each poll is very different yet you present them as if they were all the same. None of the questions are the same, there is a range of sampling approaches, the weightings that are applied are different and each pollster has its own special stamp. Would not this be better doing it for each individual pollster.
Secondly I find your linkage of actual events to changes on the chart perhaps a touch too convenient.
Rod, thank you for taking the time to prepare this interesting article. It’s clear you have a love of stats and politics and we are the richer for it.
I’m afraid, though, that since your methodology has the Tories at a 14% chance of “most seats” in a hung parliament I cannot take it seriously. Current “poll of polls” averaging has the Tories at 37.5%.
2. But Rod answered that in his last string - that was the liklihood of a Tory-largest-party situation based on the polls at the time (put another way, using a results predictor such, you’d get a likelihood of that outcome being 0%). It doesn’t predict the future, it predicts the likely parliamentary result based on given shares in a poll.
On topic, it’s a very interesting article. I have a slight concern about about the samplemiser creating nearly 600 new polls - where does the data come from? - but the results look plausible. Does it work for Lib Dem and Labour shares as well?
Of course, one of the problems with all polls and trends is that there’s no way of validating them against independent data, so all we can say is whether something feels right or not against experience and other similar polls. It would be nice if the polling firms did polls for all the scheduled local / European / London / Scottish & Welsh elections to give us a bit more information that way.
Anyway, thanks for the article which is again an excellent launching point for a debate and brings something new and useful to the table.
What is the uncertainty on the filtered data?
I was going to say what Mike S already said - that it’s far too easy to make events fit numbers in retrospect depending on personal bias. I’d like to see this for both labour and lib dems, at the moment it’s an incomplete picture and doesn’t take into account the seemingly inexorable rise of 4th parties.
A quick query -
Which polling company did you use for the data (As using all companies with their different methodologies would provide systematic rather than random bias between polls - Kalman filters are fine provided that noise is Guassian (as for polls under a single methodology) but not if multiple methodologies are lumped together)?
If they have been lumped together, may I suggest separate analyses for each company?
Do the Samplemiser people specify which Kalman filter they use?
Other than that - it’s a good idea to try, and thanks for taking the time to run the analysis.
1. The authors of the routine explain its uses across differing polls, and the concept of bias…
http://www.yale.edu/isps/publications/kalman.pdf
Basically, despite bias from polling-house to polling-house, every poll will still contain useful information. The only way to estimate for bias is to do multiple regressions BACK from an actual election. So we could do this for all polling houses back from 2005, estimate polling house bias, and assume such biases still exist to the same extent, which may not be a valid assumption…. We will only truly be able to measure current bias by comparison with the next election… I don’t have the time to estimate polling house bias using multiple regression models unfortunately, particularly as the critical sample size is hard to come by for historical polls (tsk, tsk A. Wells et al!) A reasonable working assumption is that biases largely cancel out.
I did say I would put the model to “the test.” Most of my “causes” of turning-points are speculative. It’s up to individuals to judge whether singly or collectively they are plausible.
2. thanks for your kind comments. Mike has had this article for a week or so, and it therefore does not include the most recent polls. I concede that the Tories currently stand at a little over 37%…
4. About a third of the uncertainty for any individual poll…
To answer David Herdson’s query about where the new data comes from, it’s best to think of it as the specified filter estimating what data is needed to get from A (first poll) to B (second poll, some days later) in an environment with significant noise (which has to be Gaussian - fortunately, there is a theorem that states that with enough samples, all distributions tend to Gaussian, so the “noise” on polls under a given methodology should certainly be Gaussian).
Results can be freaky but have a good real-world track record, as Rod has noted. The Kalman filter wikipedia entry (at http://en.wikipedia.org/wiki/Kalman_filter ) is an excellent summary of the filters
P.S. - I was away from the 15th to just before Xmas, and not allowed on the Net over Xmas (understandable, I guess
), so I’d like to add my best wishes for Jack W at this point.
3. Unfortunately, David, based on polls at the time, the Tories were and are always coming out ahead in most seats. Feed the mere 5% yougov advantage of the last poll into Baxter and the Tories were first. And so forth.
Rod thank you for your post at 7!
The polls in the States were certainly much more accurate this cycle of elections. But wouldn’t it be more accurate to apply a filter to them because there were so many of them? Does this method still work with the relatively few 3 or 4 valid polls per month we get from the big three?
12 Looks very interesting, Rod, so thanks again for your hard work.
I have to dash out so will re-read it and comment again later. Meanwhile my knee-jerk reaction is it looks like a tool for predicting the past - something I’m already good at.
Catch you later though.
11 - But not ahead if you fed the same Yougov figures into Wells swingometer , the Baxter proportional swing is not as realistic as the Wells UNS calculation .
11. If that was the case then fair enough. I still think there’s an argument to have these probability figures as well as the likeliest outcomes (even if you’re not convinced by the last set!).
I worked on Kalman filtering some time ago, and did some serious benchmarking of their capabilities. Much of what you say is correct, but there are a couple of provisios:
1) Kalman filters are excellent at removing “noise”, but not so good at dealing with bias. Thus, If the readings have regular or persistant bias, then the results can be very misleading. (I.e. watch out for polls that do not weight by past vote recall.)
2) The filter is effectively a moving average which gradually reduces the weight applied to older readings. Unfortunately, a single rogue reading can destabilise the Filter for a considerable time afterwards.
In summary, Garbage In, Garbage Out.
11. We’ve dismissed Baxter as useless, (without even touching on the naive boundary assumptions therein.) I’ve explained previously that leads close to change thresholds, e.g. 6% lead = Tories largest party, actually only mean a 50% chance of such an outcome…
The Kalman filter can be used for any frequency of polls from days to years…. It will always produce maximum likelihood estimates which are significantly better than individual polls or simple averages…
http://research.yale.edu/vote/tracking.pdf
A very interesting piece. I have no idea of the effort it took so I hesitate to echo the request for a version per pollster.
It sounds nice that adding the polls together “balances each of them out”, but that presupposes that there is an even amount of bias. There is not. Mori and CR push up Labour and lower the Conservatives than what actually happens in a GE. Compared to the actuals in a GE no pollster does the reverse of this. Thus there is no balance.
An additional benefit of separating the data by pollster and producing separate filtered versions is that turning points could then be compared from graph to graph - i.e. if just one pollster produces a spike or drop, it may be discounted, but if all do, then something happened then.
Further, extending the series back before the last GE (and deleting the MORI “eve-of-poll” result, which used a different methodology from the normal MORI one) can produce an idea of the systematic errors inherent in the various methodologies.
17. I can tell you it took a lot of effort(days..)
For what it’s worth, re-running the series just using the ICM, Populous and YouGov figures produced essentially the same shaped graph and turning points, with the following provisos.
The Tory trough at the end of April was a little over 34% instead of a little under 33%.
The Tory peak in June, interestingly, was a little UNDER 39% instead of a little over.
The current position would be virtually identical for LibDems and Tories, with Labour’s estimate being only about 0.5% worse on the restricted set of polls…
Really interesting. I have just picked one bit which interested me particularly and that was the presumed effect of the ‘Dave the Chameleon’ ad which you supposed dropped the Tory figure three points.
I don’t find this surprising. Despite the many prejudiced views which have been expressed here if you ask anyone in the advertising fraternity they’ll tell you that the work was outstanding. Perhaps the most effective single political ad since ‘Labour isn’t working’
To get a slogan picked up instantly would earn a creative his annual salary on it’s own. To tune into and and express a vague public perception so quickly and decisively turns it into a work of genius.
That it has ‘legs’ is for Cameron’s opponents the icing on the cake. It has already put a break on his bandwagon jumping and this is sure to intensify.
17 I agree with your comments on the need for even bias to make the results non skewed but only time will tell whether the bias is as most of us agree pro Labour with Mori and CR or in actual fact pro Conservative with Yougov et al in which case most of us will have egg on our faces .
8 - thanks, so about 1%. I’d observe that a lot of your annotations to the graph point out movements that might therefore be insignificant - but I’m not sure you too serious about these yourself.
roger I do think that the Dave the Chameleon attack ad worked for a short while, although I don’t think there have been any long term effects. I think it is seen as kind of cute, and has broadly backfired. But then as you pointed out, I am biased!
As for the filtered results, the trend appears consistent whichever filter you use. However as PtP pointed out earlier it gives an interesting story of the past. Because of the clear volatility of the data I doubt whether it can be used to predict the future to any degree of accuracy. What it does tell us is that the Tories are ahead of Labour right now, and likely to be for some time. Given everything else that is going on, I’m hardly surprised. However what it does do is give a VERY clear steer as to the real reason for the Tory revival - David Cameron. Take a look at the graph before and after. The difference is fairly accurate - a massive jump on DC’s election. I hope that ConHome will report on this because it is certainly vindication that can be used against those who argue we should change leader because we are only two or three or twelve percent ahead - those people who never voted for DC in the first place because they probably think David Davis is a bit left wing…
Rod, how do you get the data for the extra (interpolated) days from SampleMiser?
I’ve just had a play with it, and the straight output is just a smoothed version of the input data - if 60 points are sent in, 60 points are output (the ICM data I used had an output always within 0.1% of the input, interestingly enough).
Is there another tool that we’re supposed to use?
24. Andy, you have to create dummy polls for the missing days, with a share of anything (I used 34.9999%) and a sample-size of very small non-zero(I used 0.0001) These are included in the calculations, but have no effect. However, they become place-holders for the filtered and smoothed estimates for those missing days. Cumbersome, but clever, eh?
20 - Didn’t “Demon Eyes” win a couple of awards from the advertising industry? Presumably you aren’t of the opinion that everything that wins awards from “the advertising fraternity” is necessarily an indication of a campaign that succeeds for the clients?
26. It’s like some films that scored Oscars nominations (and sometimes wins)…. the fact that they’re good and win awards doesn’t mean they’re box office winners (cinema equivalent of succeeding, I suppose).
I think it’s interesting to compare this to Mike Smithson’s monthly “Poll of Polls” (just including the companies that weight by past vote).
Mike S’s “Poll of Polls” shows a much more consistent picture month on month.
I think Mike S’s “Poll of Polls” is far more credible. I have posted numerous times on this site that the vast majority of people do not follow politics at all closely - only 20% of people watch a TV news bulletin on any particular day.
Many posters on this site will never get the message but the idea that support for Parties is moving significantly following each news event is just ridiculous.
It’s true that ads won’t win awards just because they’re effective and it’s possible to win the Palme d’or at Cannes with an ad that doesn’t sell anything! Usually though award winning ads are also effective and relevant. The most ’successful’ ads at the moment are probably M+S though they have very little chance of winning anything other than an ‘industry’s own’. I doubt very much that ‘Demon Eyes’ won anything. None of the major awards have a ‘political category’and even if they did it was neither effective nor original.
Thanks for the article Rod, any chance we could see what it does for data before the 2005GE?
Also it would be interesting to see this sample meiser approach to a poll of polls.
How does it do when applied to Labour and the Lib Dems as well?
23 I am not saying you are not correct that the real reason for the Conservative revival and lead in the polls is DC and his election as leader but if a Martian was looking at the graph he may well conclude that the turning point and cause of a rise of 9% in Conservative poll ratings was all down to Walter Wolfgang !!!!
28. Well, I’ve seen Mike’s list of poll-of-polls and they are far from static, month to month… Also, you must remember they are just a snapshot, once a month. The Kalman analysis posits the changes day by day. We compared lists a couple of threads ago and the monthly snapshot figures were not that different between Mike’s p-o-p and Kalman.
RE 16, Rod, Who is this “We” who have dismissed baxter as useless?
23. I thought the turning point was John Prescott and Cherie Blair’s £7000 bad hair day which happened in the same week if I remember correctly. Overnight it made Labour look both sleazy and vulgar. Then of course Lebanon for which Blair is about to pay the ultimate political price…..
16, Due respect, but you - only you- dismissed Baxter as “just for fun” when it was showing the LDs at 6 seats, and Mike told you that Baxter has been around a long time and has a good record of accuracy.
30. If you can get me a list of polls WITH SAMPLE SIZES, I’ll give it a go.
Samplemiser IS a poll-of-polls, albeit a very clever one that tracks in time and space. It works identically for Labour and LDs, although the “shocks” (political news) seemed to fit the Tory graph particularly well…
31.”if a Martian was looking at the graph he may well conclude that the turning point and cause of a rise of 9% in Conservative poll ratings was all down to Walter Wolfgang !!!! ”
PS Rod. What have you got against Cameron?
Not so, Commentator (35). Baxter was pretty well exploded in the months leading up to the general election.
For a start, and unless things have changed, he calculates the Tory and Labour percentages, then has a fixed percentage for Others, and the Lib Dems get whatever is left over.
I suspect he has recently upped the fixed percentage to be attributed to Others, which is why there is even less than usual to be allocated to the Lib Dems.
35. I’m being gentle. It’s a discredited methodology. Do any of the following use it(or have ever used it since 1945)?
Butler
Stokes
Kavanagh
Steed
Curtice
Crewe
King
I’ll stick with them (and any sensible improvements they might make)
You go with Baxter….
re 32. Rod’s right. My average of weighted polls does show the Tory lead to have its peaks and troughs in a fairly similar pattern to the Rod chart. I suppose it is working in the same way. It is tempering ICM’s 40% Tory share this month with Populus’s 34% - both of which might be described as “noise”.
Re 16 Rod is wrong to dismiss Baxter in such a way. This is a mathematical formula for converting vote shares to possible seats at a General Election. Martin has been doing this since 1992 and has built up a good reputation. I first knew him when we were both at Cambridge and he is a mathematician of some standing.
The big argument at the moment is on how he applies the UNS with parties who are, like the Lib Dems at the moment, are seeing their poll shares decline. If the polls show that the party has lost a third of its vote since the General Election then Martin will knock a third off their share in each seat. The Wells approach would be to knock off, in this case, seven and a bit points in each seat. So in a seat where the LDs got 45% at the last election Martin would reduce that to 30% while Wells has it at 38% - hence the huge difference in projected seat numbers.
38. He’s not up to the job… an ad-man’s creation… a mere boy… being PM is a serious business, not a game, irrespective of political preference. In 10 or twenty years he might have shown his metal, but not at the moment. Given a “forced choice”, it would have to be the Dour One. Experience over youthful enthusiasm…
Re: 42, Cameron is 40 years old, hardly a boy, you sound like a Bruges group fanatic with gout. Rod fess up, you are just jealous of his youthful looks. Of course compared to the walking skull head and the other Scot, he does look young. 40 is middle aged!
43. Picked you up on previous thread if you’re about.
Re Baxter - before the last election Baxter compared his own “Transition Model” to the ordinary UNS “Additive Model” for the 4 elections from 1987 to 2001.
He found they were both almost exactly equally good at predicting the number of seats won.
See link below:
http://www.electoralcalculus.co.uk/index.html
(click on “FAQ’s - help”, “How does the model work in detail”, then see Section 4 - “Comparison and Backtesting”).
43. Don’t say that!
43. I’m 41, btw, but I stand by my judgement. Look, the Tories have taken 10 years to get themselves into this mess. It’s not my fault if it takes another 10 to get out of it…. Cameron is not their Messiah, or even their John the Baptist. I suspect come polling day many people will not take a chance on such a transparent lightweight…. > back to the thread…
43 - Come off it HF it’s us Tories who should be jealous. If only we’d had a leader with vast experience and gravitas. We too could then enjoy doing less well than at the last election up against an unpopular government and an (according to a number of Lib Dem posters) official opposition lead by a ‘lightweight’.
41 UNS is not proportional and has never been in Bitish politica even back in the 2 party days in the 1950’s . A universal swing of say 5% from party A to B was found to be roughly that in all seats rather than 8% in seats where Party A had 80% , 5% in seats where Part A had 50% and 2% in seats where Part A had 20% . Robert Waller and I have discussed possible reasons for this on here in the past .
45. But the test of a swing model is not how many individual seats are called ‘wrong’, but the net difference between the predicted and actual seat numbers for each party, based on actual vote shares rather than final polls.
41. Baxter can be good but give it Mori, CR! or to a much much lesser degree Populus and you get farcical results.
49. See 131 on Previous Thread.
50. Yes - but that is exactly what Baxter does in the section “Comparison and Backtesting” on his website!
Congrats to Rod Crosby - that’s a great piece of analysis. Possibly one of the most interesting pices on PB.com. So the Chameleon thing worked did it? How annoying of Prescott to spoil things for us.
52. Well I read it as comparing individual seat predictions, not how close it was to each party’s end result.
26 20 - Didn’t “Demon Eyes” win a couple of awards from the advertising industry? Presumably you aren’t of the opinion that everything that wins awards from “the advertising fraternity” is necessarily an indication of a campaign that succeeds for the clients?
The advert didnt fail. It could never win. People were fed up of Conservatives. Conservatives dispaired of John Major. The Media was singing to the New Labour tune.
Sometimes successes are not recognised at the time.
The fact that advertisers rekoned Demon Eyes a good advert should suggest something. In fact, it worked well as an advert because it planted a message that people are connecting with 10 years later.
If people saw it as “desparate” in ‘97. They now see it as accurate & a little bit spooky.
If Dave the Chameleon was such a run away sucess why was it so quickly dropped? I note the blog hasn’t been updated at all since September.
47 - he’s not the Messiah, he’s a very naughty boy!
54. Sorry - I was referring to your comment in post 50 about using actual votes not final polls. I should have said that Baxter’s “Comparison and Backtesting” was done using actual votes.
But you are right that Baxter’s “Comparison and Backtesting” results are looking at individual seat predictions.
57. He is the Messiah and I should know! I’ve followed a few!
Francis M: If you want to join the NuCompashinutConsurvutives, you have to really hate the Laourites.
LD Defector: I do!
FM: Oh yeah, how much?
LDD: A lot!
FM: Right, you’re in.
Or perhaps:
Dave: I’m not a Tory mum, I’m a commie, a pinko, a lefty, a Trot, I’m right-on mum, I’m a Red (C) pedestrian, and proud of it!
re 44 Punter yes I read it on the other thread “24. Quite. I think the organisational aspect is the “hidden” strength to Ming we can’t always see, though others do. Hence I could easily see a1997 scenario in that the Lib Dem vote share declines on 2005, while the number of MPs increases due to shrewder and more long term targetting.”
At these levels and trends vote share will be down to 18% and MPs probably down to 40. (Not 6!).
The challenge for the “organisation and targetting” is that the LDs are now in a squeeze. They will lose seats to Conservatives, will gain none back and have some fragile seats in Labour areas to defend where the “Iraqi voters” are believed to be moving back with Brown.
Next May almost half of Lib Dem councillors are up for election. If the LD organisation is improved then presumably that number will increase. If however it falls then do expect to lose MPs in the next GE as councillors are LDs building blocks. Reality is that LDs are 5% below the Dec 2002 polls which led into the May 03 elections. We could see a net loss of LD councillors of 200+.
58. My mistake, I thought he was going by final polls, but the main point was about net seats versus individual wrong calls.
I have no doubts about Martin Baxter’s abilities as a mathematician, and his site provides a wealth of useful information, but I don’t think that his prediction method is any improvement over UNS, and in some circumstances could end up very misleading.
62 Interesting the LDs are not doing as well as expected. Disillusioned Labour voters need a new home.
Essentially, there are 3 types of people - those that believe in Control (Socialist), those believing in Self Control (Conservative) and those believing in little Control (Liberals).
Self Control is an anathma to Socialists so, LDs should be the next best thing. Except, the BNP offer “Control” and secondly the LDs are seen as NuLabour collaborators.
62. But they got 47 with about that in 1997 a Labour landslide year. Assumin no 100+ Tory triumph and a weakening Labour Party aided by some judicious Tory tactical voting I will be v surprised if they are below say 45. RE Next May if Blair really does stick until then a mass boycott is on the cards on the likes of Hull etc by Labour activists, so even though expect a Tory tidal surge outside Cornwall in the South I can easily see ample Lib Dem gains in Labour’s North. Bromley shows how effective the Lib Dem refining is becoming. I strongly suspect they will pick and choose their battles next time ever more and more than b4, but in reality the difficulty facing the Tories is much the same hence Tory CPs in even the most nominal Lab-Lib seats will be way down the pecking order at CCHQ for resources hence my comment. Any thoughts.
BTW Do you rate Hammersmith as a realistic Tory target next time at all.
25, Rod,
Thanks for that (Been called away due to guests arriving).
I’ll have a full play with it tomorrow.
If I want separate data for separate pollsters, I’d better do some of the work myself, eh?
Thanks for the pointers, I’ll see what I can come up with.
56. “If Dave the Chameleon was such a run away sucess why was it so quickly dropped? I note the blog hasn’t been updated at all since September.”
Max, you beat me to that question. I thought that Labour’s internal polling showed the whole “Dave the chameleon” campaign had backfired?
Rod, thanks for an interesting article but I do wonder if some of the “events” mentioned are too convenient whereas other incidents like the Home Office scandals/Charles Clarke resignation and Cash for honours seem to be missing. I think that this type of political damage is more corrosive over a long period of time because of the constant drip drip of bad news. It would be harder to define the long term effects using a system like the Kalman filter?
66. The thrust of the article in descending order of importance is…
i) reducing the effective margin of error for a series of polls…
ii) identifying turning points within that series…
iii) ascribing putative causes to those turning points…
The last is clearly open to the most subjective debate. The first two are mathematically supportable…
I hope that the link works
Spread the word:
http://www.thecep.org.uk/news/ViewItem.asp?Entry=919
RE 64 Punter “But they got 47 with about that in 1997 a Labour landslide year.”
Yes but the conservative vote will be 10% higher than then!
And yes Hammersmith is a Tory win, a strong growing local base, great organisation and a weakening Labour vote.
OK - agreed. Obviously what I am about to say is speculation but I think that if the number of individual seat errors is similar then it’s highly likely that the error in each Party’s net seats will also be similar.
I think the point is that it is clear that Baxter did fully look into the merits of the 2 different methods. He is obviously a brilliantly clever person so having done this work it would be extraordinary if he was then to choose the clearly inferior method.
So my own conclusion is not that Baxter’s method is superior but that it is unlikely to be significantly inferior to the other method.
Sorry - post 70 refers to post 63.
60, 61 -
Pontius Pilate: [guard chuckles] What’s so funny about “George Osborne? ”
Centurion: Well, it’s a joke name, sir.
Pontius Pilate: I have a vewy gweat fwiend called ‘George Osborne’.
[guard chuckles]
Pontius Pilate: Silence! What is all this insolence? You will find yourself in gladiator school vewy quickly with wotten behaviour like that.
Brian: Can I go now, sir?
[slap]
Brian: Aaah! Eh.
Pontius Pilate: Wait till George Osborne hears of this!
[guard chuckles]
Pontius Pilate: Wight! Take him away!
Centurion: Oh, sir, he - he only …
Pontius Pilate: No, no. I want him fighting wabid, wild animals within a week.
Centurion: Yes, sir. Come on, you.
[takes the guard away as continues laughing histerically]
Pontius Pilate: I will not have my fwiends widiculed by the common soldiewy. - - Anybody else feel like a little… giggle… when I mention my fwiend… George…
[another guard chuckles]
Pontius Pilate: … Osborne?
[more chuckling]
Pontius Pilate: What about you? Do you find it… wisible… when I say the name… ‘George’…
[chuckle]
Pontius Pilate: … Osborne?
[both guards chuckle]
“Yes but the conservative vote will be 10% higher than then”
or more likely 3% if RodCrosby is right.
69. HF, do you think Hammersmith is a Tory win or can be a Tory win?
Went to school with one of the new Tory Lib Dem PPCs - John Barstow. Was in A-Level Politics class with him (along with two other future PPCs). Sorry to say that John was a total nutball to put it mildly. The anorak to end all anoraks. The Lib Dems are better off without him! Bet he won’t get near the A-list.
That’s two PPCs I know personally who have switched to the Tories independently.
What is the going rate these days?
74. Andrea, I wouldn’t be surprised if Hammersmith has an above average swing to the Conservatives next GE. The demographics have been moving that way for some time, and the new Conservative council seems intent on the same kind of social cleansing practised by Wandsworth and Westminster. Whether it will be enough to oust Andy Slaughter in 2009/10 is another question.
75.”That’s two PPCs I know personally who have switched to the Tories independently”
Jonathan, how many PPCs do you know overall? Should the parties keep an eye on them?
76. Kevin, if the tories are getting an overall majority, I think Hammersmith will fall.
I would certainly put it among the Conservatives potential gains, then it’s arguable if it’ll be among the “certain” gains.
I’ve the feeling I’m starting to sound pedantic!
Well, Rod, I finally got to have a good read of your Guest Slot and the many constructive comments it inspired. Well done and thanks for a fascinating piece.
My thoughts are numerous and many have already been expressed one way or another so I’ll be as brief as I can.
I think in a way the fact that the graph draws on so many many different sources is a strength, and not a weakness as Mike S in post 1 suggested. We are so critical of the various polling and similar agencies that we tend to ignore the simple truth that despite the fact they measure slightly different things in different ways, there is considerable congruity. Your filter highlights the consensus and makes is much more graspable.
There is a danger of assuming ‘post hoc ergo propter hoc’ as Mike and others (and you!) have noted. Good though your graph may be, it is useless to a stupid person interpreting it stupidly. I suggest cautiously (and I hope not stupidly) that the sharp increase in November was indeed due largely to DC, but I suspect that the April/May hike was due two something more than the behaviour of Twoshags. If B follows A, it might indeed be because of A…but you have to figure it out for yourself.
In some respects, the graph is anti-intuitive…and all the better for it. For example, I personally didn’t like the Chameleon Ad and assumed therefore it didn’t work. As Roger indicated, it probably did - and I was probably wrong.
Despite my earlier misgiving, echoed by Ben Redsell, that the tool was a predictor of the past, I now think it can help with understanding the present and therefore with calculating probable future outcomes - if used intelligently. For example, I am struck by the solidity of Tory support between November 05 and December 06. It has inhabited the 36/37 percent band pretty consistently over that 13 month period. That’s impressive and as a punter I am always impressed by solid, consistent performance as opposed to peaks and troughs. You have represented that performance in a graphic and readily comprehensible way and I will bear the image in mind when placing my election bets.
And it isn’t just useful to punters. Show it to the dorks on Conhome. If they still think DC is a bad thing for their Party, there’s no hope for them.
Don’t be hard on Baxter. It’s a useful tool which doesn’t pretend to be definitive. If we use it in conjunction with another extremely useful tool - our brains - it can be very helpful. Much the same can be said of your own addition to the toolbox and I for one am very thankful for it.
Rod
As an aside, your piece has given me the opportunity to demonstrate the value of a statistical device I have been working on. I have often noticed that the better introductory articles induce a higher quality of post and I wondered whether there was some way in which this could be quantified to calibrate the merits of the pieces introducing each thread.
I began by allocating a score to each poster on a scale of 0-10, starting with the calm and reasonable David Herdson on zero and ranging up to Mystic Moron on the maximum. Of course, David occasionally has his off days and likewise Mystic will sometimes write something sensible, if only by accident, so on a subjective basis I allow plus or minus three points according to the quality of each individual posting.
The individual scores are aggregated and divided by the total number of postings to arrive at a NPP (Nutter Per Posting) rating.
You will be pleased to note that on my provisional figures, your latest piece has scored a record low of 0.4372, the lowest NPP score of any this year, making it IMO the most successful introductory piece of 2007.
Congratulations.
….2006 even!
Happy New Year!!
80. PtP…I think that after your post I don’t look like the only one with weird “obsessions”*/interests anymore
Jonathan (75) “That’s two PPCs I know personally who have switched to the Tories independently. What is the going rate these days?”
Obvious innit really? A place on the A List and a guarantee of contesting Sutton at the next election……….
75 - I don’t think any regular Pb.com contributer is in a position to describe other people as anoraks. As to slagging off former classmates on a public forum - very classy.
Great post Red. I think the macro-shape of the graph is very interesting, it seems the Tories have big bursts and then slowly decline over 3-6 months. This suggests the Tories are still relying on “events” to keep them high in the polls.
85.”As to slagging off former classmates on a public forum - very classy”
erm, isn’t slagging off former classmates something common in UK?
I mean, I thought the point of class reunions is to get enough info about former classmates to gossip about them for the following year!
Just seen the other side to myself by flicking through ‘The Daily Mail ‘online-apparently a 20 year-old blonde page 3 girl is Dave Cameron’s latest tool to make the Tories look ‘hip’, ‘trendy’-pardon my cyncism,but the word ‘desperate ‘ is springing to mind..
87 - Not on political forums I don’t think Andrea! And as ypu know I’m to young to go to school reunions yet - have to give it at least ten years!
I’m rather surprised however that a Labour PPC thinks it acceptable to use terms like anorak and nutjob to criticise dilligent pupils though. Doesn’t sound very PC to me!
88 - And had she not been included it would have got zero publicity. I would recommend you do further research into the young lady’s excellent work. I very much doubt that any of Britains politicians could pose in a a bikini quite like she does.
Tories inventing things again, eh Max (89)? Where did you get the “diligent pupils” bit from?
The three rather confused former candidates who appear to have been seduced by Cameron´s siren songs were all below-average performers at the 2005 election. Even the one who took a second place had only 20% compared with the winning Labour candidate´s 62%. The others had 19% and 14%.
Perhaps their performance etc was something on the weak side? And they were at risk of not being re-selected? Never mind. I am sure they will all do well as the A-List Tory candidate in Sutton.
89. Max, I suppose that people don’t slagg off former classmates on political forums, because not many people went to school with people discussed in political forums!
I’m surprised you’re upset by those comments. You usually weren’t among those posters who took issue for strong terms used against politicians.
it’s not that you were afraid to post unclassy comments about politicians too, for ex the Venerable Helen (without taking in consideration she won twice in a trot. And that she’s a woman!).
The fact she wasn’t a former classmate doesn’t change the thing though.
90.” I very much doubt that any of Britains politicians could pose in a a bikini quite like she does”
http://www.espacioblog.com/myfiles/educador_13/untitled%20%20%20%20ajackson.bmp
91 - I think most people could see the inferrence.
As to inventing things I suspect the offer of a canditure in Sutton isn’t entirely accurate (like so many of your predictions). And apparently one of them had allready been re-selected so it appears you’re wrong yet again I’m afraid.
94.”And apparently one of them had allready been re-selected so it appears you’re wrong yet again I’m afraid”
who?
92 - I think it’s different if they are personally known to you. I’m related to a Lib Dem MSP and am friends with the sons and daughters of a couple of Labour MP’s. I’m sure their are things that I could say that aren’t public knowledege that would be embarrasing but I wouldn’t.
I got on with most of the people I went to school with and I wouldn’t have a go at any of them on what is a public forum.
Do you have a source for that assertion, please, Max (94)? I can´t find anything about any such re-selection on Liberal Review or on Anthony Wells excellent site.
94 - According to Pink News it was the guy in Camberwell and Peckham. Couldn’t find another source to back it up which is why I said apparently!
97 - Do you have a source for the Tories treble candidature in Sutton.
98. I don’t think the Libdems have selected in Camberwell & Peckham at all. However I’m not 100% sure.
Porter was a Southwark councillor, but he stood down this year (Labour gained his ward anyway)
96. Sorry, Max, I forgot how nice and wonderful you’re compared to nasty Labour PPCs.
None whatsoever, I´m afraid Max (99). But I can´t think of anything else which might have tempted them to switch to the Tories…….. Being on the A-List is not, in itself, sufficient, you know.
They must, after all, have had some kind of inducement to lend their names to the tosh that went out on their behalf.
100 - Very true Andrea. In many ways I see myself as the Mother Teresa of Pb.com. Spreading kindness, thoughtfulness and selflessness wherever I go.
Obviously I’m a bit younger though!
102. Max, I prefered you in your previous incarnation. However in this new incarnation, I think you would be a wonderful partner for Hazel. You should write her. Your kindness, thoughtfulness and selflessness will truly impress her.
(if it seemed that I wanted to pick up a fight with you, it was probably true…not sure why, probably other than being pedantic, I’m also irritable tonight!
103 - If you’re trying to set me up with Hazel Blears you’re definately picking a fight.
That of course isn’t to say she isn’t a beutiful, caring and warm hearted person.
O/T - And I meant to ask you, as our resident Pb.com man expert, is Elijah Wood considered good looking. A couple of people say I look like him and I’m not sure if that’s good or bad!!!!
104. well, when you mentioned the politicians in bikini, before opting for Glenda in bikini, I thought about linking you Glenda topless or CB in yfronts.
Would have been picking a fight?!
Re Elijah Wood. Personally I don’t like him, but some consider him good though
(btw, I hope I haven’t upset you when I was in the confrontational mood)
104 - I hope that’s not because you have any digusting hobbits.
Boom, boom
106. John, maybe Max is a hobbit!
86. fwiw, my take too…. I would not be surprised if by the time of the GB takeover, the Tories will be back at sub-33% in the polls….
108 - By his own admission ‘Frodo’ Max is vertically challenged
106 - Not unless you include smoking which no one in their right mind would.
105 - Don’t worry Andrea. I never find your comments upsetting. It’s one of the many benefits of being kind, gentle and understanding!
108 - I can’t believe my secrets out! I’m hoping to be the first MP to represent the Shire. I’m up against Gandolf an Ork and a Ringwraith but I’m sure I can get the electorate to believe it’s a two horse race!
109 - We shall see. My feeling is that you are governed more by ‘what ought’ rather than actually ‘what is’
111.”Don’t worry Andrea. I never find your comments upsetting. It’s one of the many benefits of being kind, gentle and understanding!”
Max, really, I’m so sorry. I was out of order.
114 - Don’t be daft. You’re about the least offensive person on here. And - between you and me - probably my favourite Pb.com contributer.
112. 25 years in analysing elections - currently pretty ambivalent between all three main parties…. If the Tories are going to make it… you’ll hear it from me FIRST….
115 - Indeed. Forelock duly tugged. But you don’t sound pretty ambivalent to Mr Cameron if I may say so.
114. Max
because I’m leaving for a few days tomorrow, otherwise I would spend the next 2 days asking you sorry every 40 minutes! 
2007 New Year Honours List
http://www.timesonline.co.uk/section/0,,5023,00.html
Some council leaders or former council leaders (like Linda Bellos who led Lambeth in the “loony left” years)
Andrea, that picture of Glenda will give me nightmares for weeks now. Gee thanks!
109. Rod I appreciate your opening article which I found intelligent and interesting. Unfortunately I think you are letting your heart cloud your judgement with that comment - GB will not put the Tories back down in the low 30 range. It appears that every time they get good coverage of DC they go up in the polls. Given the likelihood that they will be announcing policies this year - building the house on the foundations - I think the chances of good coverage are high, which will lead to stronger poll leads. The other problem for Labour and the Lib Dems is that in an Election campaign people will get more of a chance to see David Cameron, who they like, and will therefore boost the Tory vote at the critical time.
116. That’s my personal estimation of the man, and the weight of the office he seeks, and the stakes for all of us if the wrong man is chosen… (particularly at the wrong time) hint: BAD times ahead….
Would it surprise you if I told you I have only voted Conservative ONCE in my life, and that was in 1997, REPEAT 1997…. (hint: rest of the country running lemming-like in the opposite direction…)
Reason: I foresaw the coming Labour landslide, which I considered unjust and undeserved, plus I felt John Major was entitled to a “sympathy-vote”, having no control over the “hand” he was dealt in 1992….
118. What heart? I’ve yet to vote Labour in my life…..
118 Ben - “Given the likelihood that they will be announcing policies this year”
I know you mean next year, but I am still not so sure. The policy groups will report but the findings are likely to clash between groups. The actual policies will be delayed until 2008, and Brown will make the most of this weakness.
Rod,I have been away from the site for a few days and have only just read your contributions. What a waste of effort. You are clearly so obcessed with Cameron that nobody is going to take you seriously. Yet you miss the big story in British politics - the Labour transition. What a pity that in your desire to do Cameron down you fail to apply your obvious anslytical skills to the party of government.
Yeah - we all loathe Cameron but can Labour renew itself under a new leader so that once again it is an attractive proposition to the electorate? That is a question worth asking.
I am surprised that Mike gave you so much space on his site
re 74 Paul Lloyd “or more likely 3% if Rod Crosby is right.”
Since Conservatives are averaging 36% - 37% then they are 5% to 6% above the 1997 figures already.
re: 75 Andrea, Yes I see Hammersmith as a win for Tories.
re 109. How much do you want to wager on that proposition - that Tories will be down to 33% on Brown’s succession?
I’m traveling for the rest of the day but will return to the thread this evening
I know this sting’s probably ‘dead’ now, but I’ve just seen something which I think needs commenting on - namely the turning point in early October.
There were a few jocular comments above that Walter Wolfgang might have been the most significant person politically in the post-election period because of the 9 point rise that followed his ejection from the conference centre. Obviously it didn’t do Labour any favours, but equally, I doubt it said much about the Labour Party we didn’t already know.
One week later, David Cameron gave a speech to the Tory Party conference that jet-propelled him to the front of the leadership race. I think it a lot more likely that the steady rise in the vote share represents the increased expectation of him winning following the effect of that speech, than any lingering resonance from heavy-handed security officials. Many of the threads from October / November 2005 were on the leadership race at the time, and are worth checking out in that context. I think the ‘DC tipped’ comment is at least a month late: the election was all but over by then and everyone interested knew that.
My expectation is that Brown will get a small “dead cat bounce” of 2% which will come from the LDs. I expect to see the Conservative % remain in the 36%+ range (excluding Mori and CR).
But, I would expect Brown’s honeymoon to be less than 6 months as he is already well known and people have generally formed an opinion about him.
122. I simply chose the Tories because their performance is interesting(new leader’s first year) and important(determines the outcome of the next election.) The end comments about Cameron were not part of the original article and were added by Mike. I’d like to think that my personal views do not influence my objective analysis. I am not partisan to any party….
124. I’m not predicting it, just saying it’s quite possible. The original Cameron boost lasted all of 2 months, and within 5 months they were back to where they started at 33-34%. Prescott and cash-for-honours turned all this around again, but it’s pretty clear the Tory trend has been southward since late August. I’m genuinely surprised the Tories have not profited more, MUCH more, from these political “gifts”, and imho, this indicates the continuing serious weakness in their position.
125. See my comments at 68. I tried to leaven some quite heavy maths with a bit of a speculative retrospect on the year’s events. I’m quite happy for other proximate causes to be suggested, but please don’t throw the baby out with the bathwater…. Without any labels the graph tells an interesting story (to my eyes) - fragility and underlying weakness in Tory support….
127. I wasn’t meaning to throw the baby out with the bathwater. Apologies if it comes across like that. I was simply suggesting an alternative (and I think more credible), reason for the rise in Tory support from October to December last year.
As I said early on in the thread, I think this is a very useful addition the the statistical arsenal used to analyse trends.