FiveThirtyEight generally takes an inclusive attitude towards polls. Our forecast models include polls from pollsters who use traditional methods, i.e., live interviewers. And we include surveys conducted with less tested techniques, such as interactive voice response (or “robopolls”) and online panels. We don’t treat all polls equally — our models account for the methodological quality and past accuracy of each pollster — but we’ll take all the data we can get.
This split, however, between live-interview polls and everything else, is something we keep our eye on. When we launched our general election forecasts in late June, there wasn’t a big difference in the results we were getting from polls using traditional methodologies and polls using newer techniques. Now, it’s pretty clear that Hillary Clinton’s lead over Donald Trump is wider in live-telephone surveys than it is in nonlive surveys.
We don’t know exactly why live-interview polls are getting different results than other types of surveys; there are a lot of potential causes and it’s something we’ll be digging into. But it’s harder to measure this gap in the first place than you might think; pollsters make a lot of choices — whether to use a registered or likely voter sample, for example, or whether to poll Gary Johnson and Jill Stein — that could account for the differences between live and nonlive surveys. But one method to get at the difference is to use our forecast models, which account for these things.
As of Tuesday morning, Clinton led Trump by 6 percentage points and had a 79 percent chance of winning, according to our polls-only forecast. But running our polls-only model using only live-interview surveys, Clinton leads Trump by 7 points and has an 86 percent chance of winning. Running it with only nonlive-interview polls, Clinton leads Trump by 5 points and has a 71 percent chance of winning.
| CLINTON’S POLLS-ONLY FORECAST PROJECTION | |||
|---|---|---|---|
| POLL TYPE | CHANCE OF WINNING | POPULAR VOTE MARGIN | ELECTORAL VOTES |
| Live | 86.0% | +7.1 | 354.1 |
| Nonlive | 70.5 | +4.8 | 315.0 |
That’s a pretty big gap. Interestingly, about half that gap comes from the model’s trend line adjustment, which looks for how the race is changing by comparing surveys to previous editions of the same poll (e.g.., Monmouth’s August poll to its July poll.) In other words, there’s little evidence in live-interview polls that Clinton has regressed much from her post-convention high. The forecast based on nonlive surveys, by contrast, shows her convention bounce fading by a few percentage points.
The live-interview polls are much kinder to Johnson, too. His projected share of the vote jumps from 6.4 percent in the nonlive forecast to 9.1 percent in the live-interview-only forecast. Part of that gap may have to do with how some online pollsters — such as YouGov, which had Johnson at only 6 percent last week — include third-party candidates, giving respondents the option to choose “someone else.”
The live-interview/everything else gap varies a bit by state:1
| CLINTON’S PROJECTED VOTE MARGIN OVER TRUMP | |||
|---|---|---|---|
| STATE | ONLY LIVE POLLS | ONLY NONLIVE POLLS | DIFFERENCE |
| Massachusetts | +24.5 | +16.0 | -8.5 |
| Maryland | +30.6 | +22.2 | -8.3 |
| Indiana | -6.8 | -15.1 | -8.3 |
| Wisconsin | +12.0 | +4.0 | -8.0 |
| Mississippi | -5.1 | -12.9 | -7.8 |
| Michigan | +11.7 | +4.9 | -6.7 |
| Colorado | +10.0 | +4.6 | -5.4 |
| Missouri | -0.1 | -5.4 | -5.3 |
| Maine CD-1 | +17.2 | +12.0 | -5.2 |
| Georgia | +1.9 | -3.2 | -5.0 |
| North Carolina | +4.5 | 0.0 | -4.5 |
| Rhode Island | +22.5 | 18.1 | -4.4 |
| Tennessee | -9.4 | -13.8 | -4.4 |
| Texas | -6.0 | -10.4 | -4.3 |
| Minnesota | +9.7 | +5.7 | -3.9 |
| Maine | +10.6 | +6.7 | -3.9 |
| South Dakota | -9.1 | -12.9 | -3.8 |
| Pennsylvania | +7.4 | +3.6 | -3.8 |
| New Mexico | +13.1 | +9.8 | -3.3 |
| South Carolina | -1.2 | -4.5 | -3.2 |
| Alabama | -14.1 | -17.3 | -3.2 |
| Virginia | +9.9 | +6.9 | -3.0 |
| North Dakota | -10.8 | -13.3 | -2.5 |
| Iowa | +3.9 | +1.4 | -2.5 |
| Nevada | +6.6 | +4.1 | -2.5 |
| Maine CD-2 | +3.3 | +0.9 | -2.3 |
| Louisiana | -11.4 | -13.7 | -2.2 |
| District of Columbia | +77.0 | +74.8 | -2.2 |
| Washington | +16.5 | +14.3 | -2.2 |
| Oklahoma | -23.4 | -25.5 | -2.2 |
| New Hampshire | +6.0 | +3.9 | -2.1 |
| Ohio | +4.3 | +2.2 | -2.0 |
| New York | +20.9 | +19.0 | -1.8 |
| Connecticut | +13.1 | +11.5 | -1.6 |
| Florida | +4.8 | +3.3 | -1.5 |
| Hawaii | +41.2 | +40.6 | -0.6 |
| Illinois | +18.5 | +18.3 | -0.2 |
| Oregon | +11.1 | +11.0 | -0.1 |
| Alaska | -11.2 | -11.3 | 0.0 |
| New Jersey | +12.0 | +12.0 | 0.0 |
| Vermont | +25.3 | +25.8 | +0.5 |
| Kentucky | -12.3 | -10.8 | +1.5 |
| Arkansas | -14.1 | -12.6 | +1.5 |
| Arizona | -3.8 | -2.2 | +1.6 |
| Delaware | +15.4 | +17.3 | +1.9 |
| Nebraska CD-2 | -0.4 | +1.4 | +1.9 |
| Kansas | -13.7 | -10.9 | +2.9 |
| California | +22.5 | +25.7 | +3.2 |
| Idaho | -23.1 | -19.7 | +3.4 |
| Nebraska | -14.7 | -11.2 | +3.5 |
| Nebraska CD-1 | -10.5 | -6.9 | +3.5 |
| West Virginia | -20.8 | -16.9 | +3.9 |
| Montana | -9.6 | -4.6 | +5.0 |
| Nebraska CD-3 | -33.6 | -28.1 | +5.5 |
| Wyoming | -31.3 | -25.4 | +6.0 |
| Utah | -29.2 | -12.1 | +17.1 |
Clinton is forecast to win 354 electoral votes in the live-telephone-only forecast compared to 315 in the nonlive forecast. There’s not a super clear pattern in where the gap is and where it isn’t, but generally Clinton is doing better in live-interview polls in blue states and better in nonlive polls in red states. Overall, the median state in the nonlive forecast is 2 percentage points more favorable to Trump than in the live-telephone forecast. That, of course, matches what we see in the overall forecast.
But it didn’t used to be this way. On July 1, Clinton held a clear advantage over Trump no matter the mode of the poll.
| CLINTON’S POLLS-ONLY FORECAST PROJECTION | |||
|---|---|---|---|
| POLL TYPE | CHANCE OF WINNING | POPULAR VOTE MARGIN | ELECTORAL VOTES |
| Live | 80.1% | +6.3 | 343.0 |
| Nonlive | 77.5 | +6.3 | 346.2 |
She had a 78 percent chance of winning the election, according to our real polls-only forecast. And when we rerun the polls-only forecast for July 1 with only live-interview surveys, Clinton had an 80 percent chance of winning. She was expected to win 343 electoral votes and the popular vote by 6.5 percentage points. Using only nonlive interview polls on July 1, we get Clinton winning 78 percent of the time and a 6.3 percentage vote popular vote margin with 346 electoral votes.
One interesting note: Nonlive polls may have presented a false signal of competitiveness in some red states. You probably remember hearing about how Kansas and Utah might be in play? That was mostly based on nonlive telephone interview polls. A forecast of just nonlive polls had Clinton within 5 percentage points of Trump in both states on July 1. The live-interview-only forecast, in contrast, had her down double-digits in both states. Since that time, the nonlive polls have come more in line with the live-interview polls — Trump’s regained a double-digit lead in both states even in the nonlive interview forecast.2
As the cases of Utah and Kansas suggest, I’d put more faith in the live-interview polls than in other types of surveys, all else being equal. Indeed, our forecast models do just that. If the gap between live-interview polls and everything else persists, though, we’ll need to explore what might be causing the split and which is more likely to be right.