Alan Reifman’s
Party ID &Sample Weighting Page

Lettering turns grey when a more recent poll by the same organization becomes available.  All polls above are national.  For occasional discussion of partisan composition of state polls, see links section below.

(a)  I use data from the Rasmussen daily polls as my personal “gold standard” of what the “correct” partisan composition is, due to their large aggregate sample sizes and sophisticated methods:  “At Rasmussen Reports, we adjust our party identification weighting targets each month based upon actual survey results from the previous 90 days. For the month of October, our partisan weighting targets are 37.0% Democrat, 32.3% Republican, and 30.7% unaffiliated. That's little changed from our September targets of 37.0% Democrat, 32.7% Republican, and 30.2% unaffiliated” (link).  In an October 12 posting on his website, Rasmussen reports his polls have detected "no impact" of the scandal involving former Congressperson Mark Foley on party ID.  On November 1, Rasmussen announced new party ID figures of 37.7% D and 31.5% R.  Since there's not that much time until the elections, I'll just keep the October figures at the top of the chart.

(b)  For registered and “likely” voters, respectively.

(c)  Asked in terms of preference for which party to control Congress.

(d)  Among "most likely voters."

(e) Before "hard push" to see if Independents will align with either major party (or, in other polls, excluding Independents who lean D or R). 

(f) Using candidates' actual names, instead of generic reference to "Democrat" or "Republican."

My thanks to the pollsters who, in the spirit of openness and transparency, report the internal details of their surveys on their websites!

Click here to return to top of page.

Weighting Pre-Election Polls for Party Composition:
Should Pollsters Do It or Not?

by Alan Reifman

September 9, 2004 -- We have seen a lot of polls thus far in the Bush-Kerry race and we're going to see a lot more.  Often polls by two different survey outfits taken at the same time will show results in pretty stark disagreement.  Literally as I write this, Rasmussen Reports has the race a virtual dead-heat (Bush 47.5, Kerry 46.8), while CBS News has Bush up by 7.  (UPDATE:  With Election 2004 long over with, Rasmussen no longer has his daily reports up from that year; however, a 2004 week-to-week summary is still available, and the average for the week of September 9 matches closely with the daily figure I quoted on September 9, 2004.)

All pollsters try to obtain a random, representative sample of voters to represent the full electorate.  In addition to vote choice (i.e., Bush, Kerry, or other), pollsters always ask respondents which party they align themselves with.  These two measures -- candidate preference and party ID -- often show great overlap, with Republicans (R's) heavily going for Bush and Democrats (D's) heavily going for Kerry.  However, people sometimes vote for the other party's candidate, so candidate preference and party ID are not identical.

One factor (among many) that may contribute to discrepancies between different outfits' polls in their Bush-Kerry margins, I will argue, is polling firms' different philosophies as to whether it's advisable to mathematically adjust their samples -- after all the interviews have been completed -- to make the percentages of D's and R's in their survey sample match the partisan composition that is likely to be evident at the polls on Election Day.  The latter can be estimated from exit polls from previous elections, party registration figures (in states where citizens declare a party ID when registering to vote), and surveys.  

(Another issue that often comes up in evaluating pre-election surveys, with which many of you may be familiar, is whether results are reported for "registered" or "likely" voters.  That is a different issue from what is being discussed presently.  Whether a pollster reports results for registered voters, likely voters, or both, weighting by party ID is a separate, independent decision.)

Exit polls from the three previous presidential elections yield the following percentages of the electorate comprised of self-identified D's, R's, and independents (from Zogby).

A series of ten national poll readings on party affiliation over roughly the last three years from the Los Angeles Times is available here (once the page comes up, you need to scroll down a bit).  One additional source is Democratic pollster Stanley Greenberg, author of The Two Americas.  He found, after conducting 15 national polls with an aggregate 15,045 voters from late 2001 until early 2003 and allocating "leaners" to the relevant party, that each major party had the allegiance of 46% of the voters.

The controversy occurs when a poll of, say, 1,000 voters shows a partisan composition vastly different from what we've come to expect.  Should the pollster make statistical adjustments (described below) to make the party breakdown conform to more typical estimates, or should he/she just leave the numbers alone and report the findings?  A good summary of the "back and forth" of this controversy is available in this article from earlier this summer.  I will be referring back to this article.

The following two scenarios should illustrate the key issues.  Before presenting the scenarios, I want to state that there are noted national authorities on either side of the "weight/no weight" debate (including Democrats on both sides and Republicans on both sides).  Each reader should decide for him- or herself.  How to actually implement a sample weighting is described in the Appendix.

SCENARIO 1

As noted above, most recent estimates of the partisan composition of the electorate suggest a rough balance between the number of voters leaning toward the D and R parties (i.e., "50/50 nation"), with the possibility that there might be slightly more D's than R's.  

In his aforementioned book, Greenberg characterizes party identification as "... a form of social identity, not unlike ethnicity or race, with considerable durability over time" (p. 93).  I would argue that individual-level stability should generally lead to population-level stability, although not perfectly so (e.g., from one presidential election to the next, some people pass away, others newly turn 18, immigrants become eligible to vote).

Suppose a pollster completes a survey and finds far more self-identified R's in the sample than D's.  This happened in the Newsweek poll released in early September right after the R convention that gave Bush an 11-point lead over Kerry.  Newsweek's sample contained 38% R, 31% D, and 31% I (discussed here and here).  There would seem to be three plausible explanations for the higher-than-usual sampling of R's:

• There was a sudden, massive shift in party ID after the R convention.

• Given that Newsweek's polling was done on Sept. 2-3 (partially overlapping the convention), one could argue that more R's than D's would have made it a point to be home to watch the convention, thus making themselves more accessible to telephone interviewers; even after the final day of the convention, R's may have been more politically energized, making them more likely to agree to participate in a survey.

• It could have just been plain, "old fashioned" sampling error -- just as a coin, with probabilities of 50% heads and 50% tails, can yield 60% heads in a sequence of flips, random sampling of households could have yielded excessive R's just by chance.

If Greenberg is correct about the stability of individuals' party ID, then the first of the three explanations (a sudden shift) seems unlikely.  The fact that other recent polls besides Newsweek's have obtained samples with more R's than D's seems to go against the third explanation (chance).  In any event, we would conclude that if the second or third explanation were the true "culprit," Newsweek's party breakdown  would appear to be "out of whack" relative to the other aforementioned indicators.

(Again, to keep things bipartisan, the article I cited earlier as providing a good discussion of the "back and forth" of the controversy was itself prompted by a mid-summer L.A. Times poll in which it appeared there were way too many Democrats in the sample.) 

It is at this point that pollsters face the choice of whether to adjust the numbers to match more typical estimates of the D-R distribution (i.e., count R's less and D's more), or just leave the sample alone.

In 2000, pollster Scott Rasmussen went with the "leave things alone" strategy, with the result that his firm forecast a 9-point Bush victory over Gore in its final pre-election poll.  Rasmussen, to his credit, posted a candid summary on his website:

"Simply put, we had too many Republicans in our sample.  For a variety of reasons, our firm has never weighted by party.  However, if we had weighted the data before the election to include an equal number of Republicans and Democrats, we would have shown Bush leading by 2 points.  Had we weighted our data to match the partisan mix reported by the Voter News Service on Election Night, we would have shown Gore leading by a point."  (Note:  This document appears to no longer be available online.)

One would surmise that Rasmussen is probably weighting this year.  [As shown in the chart at the very top of this website, Rasmussen has indeed adopted a practice of weighting by party ID.]

Pollster John Zogby has pioneered the art of sample weighting on party ID.  Taking 1996 and 2000 together, he was the most accurate pollster in forecasting the two presidential elections.  As he notes in this essay on his website, "My polls use a party weight of 39% Democrat, 35% Republican and 26% Independent."

Another apparent sign of polls that weight is that they should exhibit less volatility day-to-day or week-to-week than polls that don't weight.

SCENARIO 2

As previewed above, a number of prominent polling authorities would presumably argue that the Newsweek poll, with its larger-than-typical R composition, or the L.A. Times poll, with the unusually wide D edge, should be left alone and not "retrofitted" into some preconceived template of what the Election Day partisan composition will look like.

According to the "back and forth" article cited above:

"Andrew Kohut of the Pew Research Center for the People and the Press said that he once conducted a survey asking voters their party twice, four days apart, and that he found substantial differences in the responses."

Further, even though Democratic candidates sometimes come out with more favorable readings on party-weighted compared to unweighted polls, Ruy Teixeira, co-author of The Emerging Democratic Majority and operator of a "blog" related to the book [moved to here], opposes weighting for party ID.  In his September 5 entry, he writes:

"Does that mean I favor polls like this weighting their samples by party ID? No, I don't, because the distribution of party ID does shift some over time and polls should be able to capture this. What I do favor is release and prominent display of sample compostions [sic] by party ID, as well as basic demographics, whenever a poll comes out. Consumers of poll data should not have to ferret out this information from obscure places--it should be given out-front by the polling organizations or sponsors themselves. Then people can use this information to make judgements [sic] about whether and to what extent they find the results of the poll plausible."

If I had to argue for not weighting on party ID, I would make two points:

• As embodied in Teixeira's quote, a "free market of ideas" should prevail.  With as much data as possible being released with polls, consumers can reach their own conclusions.  And, of course, pollsters acquire reputations over the years as to their surveys' accuracy in forecasting elections.  Over the long run, this should serve as a check and balance on polling/statistical procedures that have led them astray.

• Some pollsters who do not weight on party ID may weight on other demographic characteristics (e.g., sex, race/ethnicity), on the grounds that whether one is male or female is far more stable than whether one is a D or R.  This may help eliminate some of the skew when people appear to be represented disproportionately.

Hopefully my essay has given you something to think about as the pre-election polls roll in.  There are no definitive answers on how to handle these issues.  For all we know, Election Day 2004 may have more R's voting than D's, a departure from the last three presidential elections.  In that event, polls not weighted on party ID may end up being more accurate than those with weightings based on recent past presidential elections.   

My personal advice -- whether, to use a phrase from Senator John McCain, you're a "Republican, Democrat, Libertarian or vegetarian" -- is to interpret polls favorable to your chosen candidate in cautious terms.  If the favorable poll turns out to be accurate on Election Day, you can exult, but if it was flawed, at least you won't be blindsided.

I would like to thank John Welte for discussing sample weighting with me over the years.

Click here to return to top of page.

Click here to return to top of page.

Additional Polling Resources 

Polling Report (compilation of poll results)

Performance of different survey outfits' polls in forecasting true presidential vote from 1936-2000 (and 2004)

Letter by Dr. Reifman in the ISR Sampler (published by the University of Michigan's Institute for Social Research), arguing that the older concept of "quota sampling" and sample weighting are conceptually similar (when the document opens up, go to page 12 to see Dr. Reifman's letter and responses by two University of Michigan survey experts).

Statement by polling firm Survey USA on its sample-weighting policy (from Daily Kos)

USA Today article on the sample-weighting controversy (original "Move On" advertisement to which Gallup is responding in article)

Pollster.com (formerly "Mystery Pollster"), a blog devoted to polling methodology issues, including sample weighting by party ID

Political Arithmetik (quantitative analyses of polls by Wisconsin Prof. Charles Franklin, who taught Dr. Reifman in a summer stats course at Michigan over 20 years ago!)

Party ID in the USA, 1952-2004, from the University of Michigan's National Election Study (gets at the question of the stability of party ID)

Treasure trove of 2000 national and state-by-state exit-poll data (from MSNBC), if anyone wants to compare partisan composition in any new 2004 polls with earlier benchmarks [you need to scroll midway down on right-hand side for menu to select national or particular state's data, then scroll way down on left for actual data]

Letter to the editor by Dr. Reifman, published in the Wisconsin State Journal (Dr. Reifman's letter is about halfway down in the set of letters, under the heading "Polls may be weighted") 

"A Consumer's Guide to the Polls" (2004), reviews the practices of the leading polling outlets on several dimensions

Bloggers Chris Bowers and Steve Soto were very active in analyzing the sample-weighting controversy in 2004.  Their writings led me to many of the websites linked above, for which I thank them.

Academic article (Erikson et al., 2004, Public Opinion Quarterly) on the volatility of samples' partisan composition in daily tracking polls.

Daily Kos diary by Dr. Reifman on the partisan composition of a poll in the New Jersey U.S. Senate race, with Kos reader comments (9/25/06).

Memo by pollster David Winston on possible oversampling of Democrats in 2006 pre-election polls  (10/16/06).

Website of Columbia University professor Andrew Gelman (see his 2001 article on post-stratification, in the Journal of the American Statistical Association, as well as other related articles)

Click here to return to top of page. 

APPENDIX:  Implementing Sample Weights in the SPSS Program

As shown in the grid below, I have created a fictitious data set of 30 respondents.  I made it so that 10 Democrats were sampled (80% of them voting for Kerry, in accordance with actual survey estimates), along with 20 Republicans (with 90% for Bush).  The gray shading is simply meant to break the grid into blocks of five lines, making it easier to count cases.  

If no weighting by party were done, Bush would be leading 66%-33% (20 to 10 in raw numbers).  

If we want to re-estimate the sample with 50% Republicans and 50% Democrats (which roughly matches some actual estimates), then we would weight each Democrat 1.50 (to bring them from 10 respondents to 15) and weight each Republican .75 (to bring them from 20 respondents to 15).  

The weights should be treated like any other variable, with a name such as "pweight" for party weight.  In SPSS, you would go to the "Data" menu, then "Weight Cases," then weight by pweight.  

With the weighting implemented, the sample becomes equalized on numbers of D's and R's, and Bush's lead shrinks to 55%-45%.  (The easiest way to visualize this is to aggregate the weights of the original Kerry voters [8 X 1.50] + [2 X .75] = 13.50, which is then divided by 30, yielding .45.)