Fisking a Poll

My son sent me this Pew Research Center poll which makes the claim that Obama has a bigger lead than anyone since Bill Clinton in 1992 and '96. 

At this stage in the campaign, Barack Obama is in a strong position compared with past victorious presidential candidates. With an eight-point lead over Mitt Romney among likely voters, Obama holds a bigger September lead than the last three candidates who went on to win in November, including Obama four years ago. In elections since 1988, only Bill Clinton, in 1992 and 1996, entered the fall with a larger advantage. 

Not only does Obama enjoy a substantial lead in the horserace, he tops Romney on a number of key dimensions. His support is stronger than his rival’s, and is positive rather than negative. Mitt Romney’s backers are more ardent than they were pre-convention, but are still not as enthusiastic as Obama’s. ...

Since this flies in the face of what most of us think is going on, I thought I'd look at this a little more closely and see if I could find anything fishy about their statistics or their approach.  Read the fine print.  While they don't exactly hide this information they put it at the very back of the report, page 6. 

One of the most pervasive problems with polls is bias in the selection of the group polled.  They explain their methodology - kudos to them for that - and present us with some descriptive stats on who they think they polled.  That will be my starting point:

First thing to examine is the fact that there are more "registered voters" than "likely voters".  Likely voters are the preferred people to poll because they're interested enough in the process to know they're going to the polls.  It's always turnout that matters.  Most Stupid party folks I know would walk through fire to vote Obama out; are the Evil party folks that motivated?  Some are, for sure, but more than Stupids?  Unfortunately it's hard to hang a number or correction on this value (other than that 10.5% more registered voters than likely voters were polled, which isn't much).

Among the registered voters they polled 21% more Democrats than Republicans.  This is a big red flag.   While we can't completely rule out "Obama Republicans" (today's version of Reagan Democrats), and we can't rule out Romney Democrats, all things being equal, you'd expect the groups to be within a percent or two of party lines.  They also poll more independents than Republicans by a more modest 5%.  Independents, of course, are what that whole 47% thing from Romney was all about - that he had to put his effort on the undecideds, not the ones who are completely in the bag for Obama and won't change (which was all blown out of proportion by the lame stream media). 

Another red flag is that they had 11.9% more people who said they were Obama supporters than Romney supporters.

In summary, this poll is biased as all hell.  In a country where a 4% win, like 52%-48%, is a big win, they start between 12 and 20 percentage points in favor of Obama.  This poll can't possibly be accurately measuring the electorate.  Actually, given that 12-20%, he should have done better in this poll than the 8% they claim.  That probably augers badly for Obama. 

So how should the sample group be composed?  Does it need to be 50-50?  Rasmussen has an interesting page showing the party affiliations by month every year.  This August (still the most recent) breaks down as Rep: 37.6%, Dems 33.3% Other 29.2%, which is 4.3 more registered Rs than Ds (R-D).  In August of 2008, it was 33.2% Rs, 38.9% Ds and 28.0% others, for a -5.7% R-D.   That's 10 point swing in registered Republicans vs. Democrats compared to August of '08.  

That implies the polls should over sample Republicans over Democrats by at least the current 4.3% more registered (if not the 10% difference from '08 to now) to better model the electorate, but none of them that I've seen do that.

I debated with myself whether or not to put this simple an analysis up for my readers.  Most of you are probably better at this than I am.  But you never know who's going to be wandering around the Inter-Toobs and a little education is a good thing.  It's not hard.  All you need to do is percentages on a calculator.  No p values, no r coefficients, no confidence limits.  Honestly, I've never seen a poll that needed fancy statistical techniques to evaluate.  It's easy to decide whether to believe it or not.  But, as the famous guy once said, "trust, but verify".  Do your own math.