A 2016 Election Post-Mortem: The ABC News/Washington Post Tracking Poll Presented at the annual meeting of the American Association for Public Opinion Research New Orleans, LA May 14, 2016 Gregory Holyk, Gary Langer - Langer Research Associates Scott Clement - The Washington Post
25
Embed
A 2016 Election Post-Mortem: The ABC News/Washington Post Tracking Poll
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A 2016 Election Post-Mortem:
The ABC News/Washington Post
Tracking Poll
Presented at the annual meeting
of the American Association for Public Opinion Research
New Orleans, LA
May 14, 2016
Gregory Holyk, Gary Langer - Langer Research Associates
Scott Clement - The Washington Post
Credibil ity of the polling industry has been questioned
following Trump’s surprise victory.
We performed our own post-mortem of the ABC
News/Washington Post tracking poll.
Abt Associates
Dual-overlapping RDD, English/Spanish, 65% cell
18 daily waves, Oct. 22-Nov.6
Daily n=440, increased to 800 for the last 4 days
Each estimate (except the first release) based on ~1,100 -
1,200 likely voters, increasing the last 4 days to peak at
2,200 for the final estimate
Fallout From the 2016 Election
Final popular vote estimate was 2 pts. off, exactly the mean
error in final estimate going back to 1984.
Dug around for any possible irregularities and found:
LV modeling made sense
No indications of differential non-response by groups
No improvements using different weighting procedures
No evidence of “shy” Trump voters
No consistent interviewer effects
At the end of the day, the overall narrative based on state -
level polls and aggregator probabilit ies was off.
As the AAPOR report concluded, there was no major failure
of high-quality national polls, including the ABC/Post
tracking poll.
Key Takeaways
Vote Preference Trendline
Oct. 20-22 data showed 50-38% Clinton-Trump
At the time, this estimate seemed reasonable given:
The Access Hollywood tape had come out recently
Many Republicans publically pulled their support
He was widely seen as having lost the debates to Clinton
Enthusiasm among Trump voters was down significantly
The share of registered leaned Republicans who were certain to
vote also was down
Narrowed quickly in the next 5 days as new data was rolled
into the estimates (+12, +9, +6, +4, +2, +1)
Shifts l ike this occurred in 1996 and 1992 as well .
Variation in Tracking
ABC/Post
final est.
Actual
vote
Major party
margin error
Major party
total error
2016 47-43% 48-46% 2 pts. 4 pts.
2012 50-47 51-47 1 1
2008 53-44 53-46 2 2
2004 48-49 48-51 2 2
2000 45-48 48-48 3 3
1996 51-39 49-41 4 4
1992 44-37 43-37 1 1
1988 44-52 46-53 1 3
1984 40-55 41-59 3 5
Average 2 3
ABC/Post Past Performance
Predictive Accuracy of Final Polls
Evaluating the Tracking Poll
Constructed more than 30 cut-off LV models
Estimated turnout:
ABC/Post LV model 61%
Actual turnout, VEP (highest office) 59%
Actual turnout, VAP (highest office) 55%
Nearly all of the models showed a +3-5-point Clinton lead.
Evaluated a regression-based probabilistic LV model (only
possible in mid-tracking) and it moved estimates 2 pts. on
the margin toward Trump.
Will explore this approach in the future.
Advantage: Uses all respondents in the vote estimate, weighted to
their likelihood of voting.
Likely Voter Modeling
Unlike many state-level polls, we properly weighted for
education, as we always do, so that wasn’t a problem.
Did the weight need to be adjusted for population density?
We calculated a new weight that took into account
population density.
This had no notable effect.
Is there evidence of shy Trump voters among those who
participated in the survey? No, those who refused or DK’d
the vote question didn’t look disproportionately like Trump
voters.
Weighting
Perhaps support for Trump was underestimated because the l ikely voter models generally excluded those who were not certain to vote?
No. In waves where we asked vote preference among probable, there was no shift towards Trump.
The sample of likely voters looked very similar in both cases.
Did those who refused to answer the vote question end up supporting Trump?
It’s not clear this would have helped Trump.
Refusers were more apt to be women and minorities, but also less likely to have a college degree and to be younger. (Those who DK’d also were less apt to have a college degree.)
Among refusers, leaned Democrats outnumbered leaned Republicans.
“Shy” Trump Voters
Pre-election polls often overestimate vote for third party
candidates
This election had two major third party candidates in
Johnson and Stein
Our final estimate didn’t miss the third party candidates by
much:
4 percent for Johnson in ABC/Post, vs. 3 percent actual vote
1 percent for Stein in ABC/Post, vs. 1 percent actual vote
Johnson voters in our poll looked a lot closer in
demographic characteristics to Trump voters than Clinton
voters, but we were only over by 1 pt.
Third-Party Misestimation?
How Did Tracking Compare to
Other Indicators?
No clear benchmark for turnout and vote preferences among
groups.
Exit polls have problems:
They’re weighted to actual vote, but cannot be adjusted for size of
group since that’s unknown.
Research suggests exit polls underestimate less-educated,
younger and nonwhite voters.
Best comparison for the tracking poll is to multi level
regression with poststratification (MRP) estimates.
Comparing Tracking, MRP and Exit Poll
Tracking and MRP Turnout and Vote
Final tracking Final MRP
Size Cl-Tr Diff. Size Cl-Tr Diff.
All 100% 47-43 +4 100% 47-44 +3
Whites 72 37-53 -16 73 38-54 -16
No degree 59 42-49 -7 63 42-49 -7
Degree 41 54-35 +19 37 54-36 +18
Among whites:
Men, no deg. 18 29-60 -31 21 25-65 -40
Women, no deg. 23 29-64 -35 24 35-58 -23
Men, degree 15 43-42 +1 14 43-46 -3
Women, degree 15 51-39 +12 14 55-37 +18
MRP Turnout Predictions
MRP Margin Prediction Errors
MRP in the Swing States
HuffPost Pollster MRP Estimates
State Actual Average Error Predicted Error State N
Georgia -5.2 -2.4 2.8 -5.6 -0.4 332
Wisconsin -0.8 6.1 6.9 -0.2 0.6 169
Florida -1.2 1.8 3.0 -0.5 0.7 658
Michigan -0.2 6.0 6.2 0.5 0.7 291
Pennsylvania -0.7 4.1 4.8 -2.7 -2.0 375
Colorado 4.9 4.9 0.0 7.0 2.1 139
Minnesota 1.5 6.9 5.4 3.7 2.2 163
Arizona -3.5 -1.6 1.9 -5.8 -2.2 214
Virginia 5.3 5.3 0.0 2.9 -2.4 277
North Carolina -3.7 1.6 5.3 -6.1 -2.5 323
New Hampshire 0.4 3.3 2.9 3.2 2.8 38
Ohio -8.1 -1.0 7.1 -5.2 2.9 367
Iowa -9.4 -3.0 6.4 -5.0 4.4 99
Nevada 2.4 2.1 -0.3 7.0 4.6 81
Our MRP 538 HuffPo DKos
YouGov
MRP
Clinton % 46.8% 48.5% 45.7% NA 47.9%
Trump % 44.3% 44.9% 40.8% NA 44.1%
Margin 2.5 pts. 3.6 pts. 4.9 pts. NA 3.8 pts.
Correct predictions 50 46 46 46 43
RMSE margin all 5.8% 7.1% 7.1% 7.0% 7.6%
RMSE margin no AK, HI, DC 4.5% 6.7% 7.2% 7.0% 7.6%