In-Sight Publishing

Author(s): Scott Douglas Jacobsen

Publication (Outlet/Website): The Good Men Project

Publication Date (yyyy/mm/dd): 2024/10/29

Carl Allen discusses correcting misunderstandings about polling, particularly the common belief that polls predict election outcomes. Allen highlights that polls are snapshots of current opinion, not forecasts, and that even experts often misinterpret them. He compares polling to theoretical ideal gases, emphasizing that no poll is perfect, even in optimal conditions. While there’s “no such thing” as an ideal poll, there’s also no such thing as an ideal gas: but the “ideal” framework allows for a set of standards any poll, even nonideal ones, can be compared to. Currently, many people are shocked to find, that analysts in different countries use different, contradictory standards based on nothing more than their country’s tradition. Because polls are currently so poorly understood, Allen has drawn upon easily understood examples and analogies that are technical enough to satisfy any expert, but not too technical for the average reader. The book addresses issues like misinterpreting poll margins, the role of undecided voters, and how even well-conducted polls can be misused by media and analysts. Allen advocates for more transparent methodologies and scientific rigor in polling analysis. 

Rick G. Rosner is known for high scores on various high-range tests. He earned 12 years of college credit in under a year and graduated with the equivalent of eight majors. Rosner has written for popular shows like Crank Yankers and Jimmy Kimmel Live! and won a Writers Guild Award. He’s also worked as a bouncer, stripper, and roller-skating waiter. Featured in Errol Morris’s First Person, Rosner lost on Jeopardy! and famously sued Who Wants to Be a Millionaire. He lives in Los Angeles with his wife and two dogs.

Carl Allen is the author of The Polls Weren’t Wrong and the owner of Triple Digit Speed Pitch, LLC. With a background in sports and political data, he has created new polling metrics and previously worked as a data scout for MLB and NFL. Allen holds a Master’s degree in Sport and Fitness Administration from the University of Louisville and is bilingual in English and Spanish. He is also a  passionate advocate for statistical literacy in polling analysis and science as a whole

Scott Douglas Jacobsen: Today, we are here with ‘RealRickRosner’ and ‘RealCarlAllen’ to talk about polls, particularly a new book by Carl. The new book’s title is The Polls Weren’t Wrong, by Carl Allen. Thank you very much. Let’s take a step back. Carl, what was your original motivation for writing this particular book?

Carl Allen: I saw so much unscientific, poor analysis being done—not just in the media, but at the very top of the field. Often, when we have a complex concept that only scientists understand, the media breaks it down in a way that makes you cringe a little. It doesn’t make sense.

Polling data is unique at this point in the field. It’s been around for nearly 100 years in political polling. However, the problems with how the media discusses polls—the same misunderstandings—exist at the very top of the field. I always tell people why my book came out in 2024 and not 2020 because I couldn’t convince myself that people with the credentials—the experts in the field—didn’t understand some of these basic concepts. One of the most basic concepts is that polls are not predictions.

Polls are not predictions of election outcomes. When I say that, some people nod, saying, “Yes, that makes sense.” Others are extremely skeptical of the idea that experts—those who publish and write academic articles and books—don’t understand that. So, I had to show that they believe an accurate poll will predict an election outcome. That is a fundamental misunderstanding of what polls do and the data a poll provides.

I approached this not as a scientist or statistician but as a researcher and educator. The purpose of the book, in short, is twofold. First, it is to inform the public. Right now, the public needs to be more informed about how polls work, and that’s a problem.

Second, and slightly harder but arguably more important in the short term, is to fix how experts analyze polls. If experts change how they interpret and explain polls, the media will follow, and the public will be better informed. But right now, misinformation runs throughout. One of the main points of Carl’s book is that you can have excellent poll results. Still, the media and even high-level analysts can misinterpret the results.

Rick Rosner: And I have a lot of other gripes because most polls aren’t perfect.

Allen: No poll is perfect. There’s no such thing as a perfect poll. But as scientists, one way we explain things to beginners is to ask, “If this measurement were perfect in every way, what would it show us?” One example I use in the book is ideal gases.

If you’ve taken a chemistry class, you might remember ideal gases: P.V. equals nRT, volume, and temperature, all of which you must account for. But the secret about ideal gases is that they aren’t real. It’s not a real phenomenon. It’s something that educators and scientists created to explain a much more complex concept. Political polls, however, are easier to understand than thermodynamics.

While theoretical ideal polls and ideal gases are easier to comprehend, ideal polls are much easier to grasp. In my first presentation to the reader, I demonstrate what an ideal poll measures and show that even if a poll is conducted perfectly in every possible way, it would still result in imperfect data—an imperfect estimate. This understanding is crucial: even an ideal poll can be imperfect.

We shouldn’t have higher expectations for real-world, non-ideal polls than ideal ones. 

Rosner: Can I give you some real-world examples? We do have a couple. Last week, two polls showed crosstabs for people aged 18 to 29. For context, a crosstab is a subset of data for a specific demographic group. In this case, young people aged 18 to 29. One poll had Kamala Harris up by 31 points. Another poll, released within an hour of the first one, had her up by just 3 points.

So, what do you do with that? And how does that happen? That’s a real-world example, number one. Another example from today involves two candidates for governor in North Carolina: Josh Stein and the flawed Mark Robinson. Last week, Robinson got into trouble—he’s always posted lunatic stuff—but this time, new information surfaced about his past. He had been posting on a site called “New to Africa,” discussing inappropriate sexual encounters with his sister-in-law and boasting about them. He also called himself a “black Nazi” and said if we could still enslave people, he would own some. He’s a jackass.

The first polls measuring the impact of this scandal came out, showing Stein leading Robinson 50 to 35.  Compare that to the polls for Harris versus Trump, which often show Harris leading by 52–48 or 51–49. One notable difference between these polls is that the numbers add up to 100% in the Harris examples. That could be a good sign, or it could be misleading. Did they remove the people who expressed no preference?

But the Stein-Robinson poll only adds up to 85%. In the book, you explain how the missing 15% can cause various issues and lead to swampy conclusions.

Allen: Absolutely. That’s a perfect point. When the numbers don’t add up to 100%, it indicates that some people gave a response that needs to be reflected in the top two results. To simplify, Stein was at 50%, and Robinson was at 40%.

50 plus 40 is 90. That means 10% are somewhere else. Where is that somewhere else? In some elections, part of that 10% expresses support for a third party. Only I don’t believe any third-party candidates are running for governor in North Carolina. If there are, or there are “write-ins” I doubt they get much of a percentage.

Allen: It’s Democrat plus Republican, equaling 100%. That’s the only possible outcome. It’s binary. Election results are binary. In this case, it’s a simpler example—Democrat plus Republican equals 100%. There is no other possible outcome. But before the election, even with only two candidates, there are still three possible responses in the poll: Democrat, Republican, or “don’t know yet” (or undecided, depending on how you split it).

The flawed logic comes in when analysts try to eliminate the undecided option, attempting to make an apples-to-apples comparison. So, let’s say Stein (the Democrat) is at 50, and Robinson (the Republican) is at 40. We know that 50 plus 40 isn’t going to be the election result. So, where do those other 10% go? They’re in the land of the undecided. The poll doesn’t tell us how those people will eventually vote—it only tells us how many there are.

As a forecaster, I make a prediction when I dig into the data. My job as an analyst is to figure out where the undecided are most likely to go. In the U.S., the assumption is that undecideds will split 50/50. That assumption feels reasonable and safe, but the reality is that it rarely happens. We know this from past elections and past data.

Even intuitively, a 50/50 split might only sometimes occur. So, when we account for the undecideds, I say, “If they split 60/40, we should account for that. If they split 70/30, we should account for that. If they split 50/50, we need data to support that.” We cannot just assume that a 50/50 split is the default or the null hypothesis.

That is not valid science. In the U.S., instead of saying “50/40 with 10 percent undecided,” they often say “50/40, so this candidate is up by 10.” They assume the election result must be that Candidate A wins by 10. They assume the undecideds will split evenly, 55/45. The unscientific part that misinforms the public and that I’m trying to correct—is the assertion that any discrepancy from a 55/45 result means the poll was wrong.

They claim that if a poll shows a 50/40/10 split, it means Candidate A must win by 10 points in the election or else the poll was inaccurate. That’s the entire logic. I’m not exaggerating or taking liberties here. This is the academic definition—universally accepted in the United States—of poll accuracy: the margin, or difference between the two candidates in the poll, must equal the margin in the election result, or else the poll is considered wrong.

Rosner: And we talked about this a couple of days ago—people have a problem with uncertainty.

Allen: The entire gambling industry is based on people believing they can predict something highly unpredictable, like the outcome of a sports game.

Rosner: Even Einstein, the smartest guy in the world at the time, couldn’t handle quantum mechanics. He said that some things are inherently unpredictable. He spent so much of his later life trying to figure out how that could be, and he famously said, “God does not play dice with the universe.”

Allen: I was just about to give the same quote. Yes, that’s a great quote.

Rosner: So, people want certainty, and those who analyze polls want to give the illusion of predictive certainty. This leads people into all sorts of confusion. In the case of Robinson, there’s another confounding factor: voters who are so discouraged by their candidate that they don’t even tick a box. They may vote in the presidential election but not in the governor’s race. That could happen, and it does happen. It happens both ways. There will also be voters so disappointed by the presidential candidates that they only vote for the down-ballot candidates.

Allen: All of this makes sense. These things will happen in unpredictable numbers, making it even tougher to make any reliable prediction. I love the dice analogy because fair dice have a known probability. We can calculate the outcome of rolling two fair dice with extreme precision.

One of the issues I have with statistical literacy is that statistics classes often focus on precision. They teach students to calculate probabilities down to the decimal point, which is fine. There’s nothing wrong with that. But more basic statistical literacy, which I emphasize, involves asking: even if you can’t calculate the exact probability, can you still tell me which outcome is more likely?

For example, when rolling two fair dice, the sum of the dice will always be between 2 and 12. The most common outcome is seven because there are more ways to roll a seven than any other number. More combinations add up to 7. It’s the most common outcome but still not likely—just more probable than the others.

Rosner: There’s only about a 17% chance of rolling a 7—6 out of 36 possibilities. 

Allen: For some learners, this helps them wrap their heads around probability. For others, it doesn’t click as easily. When I was talking to a high school stats class the other day, here’s the example I gave:

I said, “There’s a prize if you’re on the winning team. I won’t tell you the prize, but Team 1 wins if the dice sum is 2, 3, 4, or 5. Team 2 wins if the sum is 6, 7, 8, 9, 10, 11, or 12. Pick your teams.”

Every single student in the class—all 30 of them—chose Team 2 (6, 7, 8, 9, 10, 11, 12). So, I asked, “What are the chances that your team wins if I roll the dice?” The students started pulling out their calculators.

I said, “No calculators. Just give me a rough guess.” 

Rosner: 6 out of 36.

Allen: I wanted them to give me an approximation.

AllenRosner: One of the students said 80%, another said 60%. I responded, “But you both said greater than 50%, so your side is favoured. You can say that with certainty.”

I asked, “Are you 100% certain that your sidete is favoured?” They looked at each other briefly and said, “Yes, I’m 100% certain that our side is favoured.” So I continued, “You don’t know if the probability is 60% or 80%, but you’re certain it’s greater than 50%?” They all nodded in agreement.

I said, “Beautiful. This is an excellent lesson in probability. Even though you don’t know the exact probability, you can still say with certainty that it’s greater than something. In this case, greater than 50%. Now, we can calculate the probability with certainty: it’s 26 out of 36, about 73%. But here’s the key—while all of you chose the same option and agreed that this side is favoured, is it possible you’re wrong?”

The word ‘wrong’ carries a misleading meaning. Does “wrong” mean you chose the wrong favourite, or does “wrong”, mean the result doesn’t match what you predicted? This is where we get into the concept of The Polls Weren’t Wrong, saying that polls are not predictions. Polls are not predictions of election outcomes. Polls are observations of the present state.

All the students correctly observed that they’d go with 6, 7, 8, 9, 10, 11, or 12 if they had to choose. I rolled the dice, and it was a 9. Everyone won. Woo! Everybody was happy. But we know mathematically, with certainty, that they would have been “wrong” about 27% of the time.

Rosner: Exactly, and this brings us to the 2016 election. That leads almost directly into it because some forecasters said Trump had a 66% chance of losing, while others said 99%. Can we say with certainty that the 99% estimate was a badnumber? Yes, we can—but that takes a few steps to explain.

When discussing discomfort with uncertainty, people tend to remain rational when discussing dice or coin flips—things with clear, calculable probabilities. But emotions run high when we start talking about political polls and elections. Anxiety spikes, especially around 2024, as Trump does nothing to calm people’s fears about what his second presidency might look like.

Allen: Absolutely. To tie this into the book and build on what we’ve been discussing, Chapter 4 introduces the concept of ideal polls. If a poll is conducted perfectly, by every possible measure, this is the data it would produce. Many of the reviewers of my book said, “But this isn’t possible.” My response was, “A: That’s bullshit. It is possible, and I can prove it. B: Even if it weren’t possible, that doesn’t mean it’s not a useful concept.”

In the same way, chemistry classes teach about ideal gases or optometry, and physics classes teach about ideal lenses. Ideal lenses aren’t real, and ideal gases and fluids aren’t real, either. But we use these concepts to understand what would happen under perfect, ideal conditions.

A political poll can never be conducted without some margin of error because of the margin of error. That’s a statistical fact. It doesn’t matter what your sample size is. If you take a census, you’re no longer conducting a poll.

Rosner: A census is where you get the actual result from every participant in what you’re trying to measure—in this case, the population. 

Allen: Exactly. And when we use the word “population” in statistics, we talk about the people of interest or the measured items. 

Rosner: In the context of elections, the votes cast are, by definition, a census because you’re capturing data from everyone who voted.

Allen: A census of actual voters is simply the election results. We often deal with tens of millions, sometimes even over 100 million, when discussing election numbers. But our brains aren’t built to handle numbers that large. We’re wired to think in terms of tens or hundreds. So, when I introduce the concept of ideal polls in Chapter 2 of the book, I explain how sample size works to achieve a given margin of error. The sample size that tends to be used is around 600.

Rosner: After a certain sample size, the size of the population no longer matters.

Allen: This is an unintuitive fact of statistics, but it’s testable and provable. Students often see the margin of error as an abstract concept, just numbers plugged into a formula. However, my approach is to show that even experts often misunderstand and need to understand what the margin of error applies to.

There needs to be more clarity on what this formula means and how it’s understood. I provide several examples in the book, but here’s a simple one: Imagine you’re in a class of 100 students. You can ask them whether they have a dog or what they plan to eat for lunch tomorrow. You’d need to survey about 86 students to get an accurate sample – a margin of error down to +/- 4%

So, I took a random sample of 86 students and asked them questions. Let’s say I get 50% for option A and 50% for option B. What do those numbers mean? It means that if I had asked every student in the population the same question simultaneously (instead of only the same of 86), then the results would be within the margin of error as often as dictated by the sample size and the confidence interval.

This is a very important point: the margin of error relies on the subjunctive—on the “if” statement. It’s hypothetical in nature. If I had asked everyone, the numbers would be within the margin of error. This is a testable, provable concept. The math behind the margin of error works because of this “if” scenario. That’s critical to understand because it explains how polls are tools for observation, not prediction.

I always emphasize this when explaining statistical literacy. You don’t need to understand every formula in depth, but you need to grasp the concept that underlies the margin of error—it’s about the likelihood of the results being representative if you had surveyed the entire population. That’s the essence of how the margin of error works, which I tell students often.

If you’re decent at coding or Excel, you can simulate polling with a population of a million A’s and B’s, then take a random sample of 600 to 1,000. Ninety-five percent of the time, you’ll get results within the confidence interval. 

Rosner: But people on T.V., even experts like Steve Kornacki, only sometimes consider that when working with numbers on their boards. They’re thinking in terms of spreads and margins. When we look at a poll that says 50% for Candidate A, it doesn’t mean 50%.

Allen: It means 50%, plus or minus 3%. The example I give in the book—and it’s great because it perfectly illustrates the point—is with dice. If you ask me for a 95% confidence interval for the roll of two dice, my answer would be 7, plus or minus 4. That means I’m 95% confident the outcome will be between 3 and 11. You eliminate the extreme possibilities, like 2 or 12. You’re 95% confident that the dice roll won’t be a 2 or 12.

So, saying “7 plus or minus 4” means the result could fall anywhere within that range. Imagine the misunderstanding if someone said, “Carl said 7.” No, I didn’t say 7—I said 7, plus or minus 4. Those are very different statements, and the same misunderstanding happens with polls. When a poll says 50%, it means 50%, plus or minus 3. The number RIGHT NOW is likely to fall somewhere in that range. But as we discussed earlier, that 10% undecided can and will influence the eventual result – the election. Different calculations.

Rosner: That’s a great point. But let’s shift to some real-world gripes. In the book, you ask innocuous questions to people, like, “What are you doing for lunch?” Most people will answer that question. Some might say, “None of your business,” but you’ll get a decent response rate. But consider this: The New York Times/Siena poll made 194,000 phone calls to get 2,000 respondents, meaning only one out of 100 agreed to participate. That raises the likelihood that some people are not representative and may even be fringe respondents.

On top of that, this election cycle has an added layer of deception. I suspect—and this is just a theory—that some Trump supporters might purposely give false answers. For example, a MAGA voter might say they’re a Democrat voting for Trump to manipulate the poll results. It looks more significant when a Democrat switches to vote for Trump.

I agree that’s possible, though there’s no way to prove it definitively. I think that potential issue might have lessened when Harris replaced Biden as the Democratic candidate. But as you said, all of this is speculation.

Allen: While it could happen, the impact would have to be large and well-orchestrated to make a significant difference in the poll results. 

Rosner: It could shift the numbers by 1% or 2%, no question; however, in an election where 1% or 2% could be the margin in key states, even that small shift can have a big effect.

Allen: That small margin can make all the difference in close races. Still, the likelihood of large-scale coordinated false responses is low. It is. So it’s very hard to detect that small movement and change. I want to add something, too. I noted your earlier speech when you said that people want poll data to be predictive. In my conversations with pollsters, it often comes back to what they believe people want to see. Remember, pollsters usually don’t pay out of their pockets to conduct polls. Most of the time, they are funded by media outlets or sponsors. So, one of the things I’ve had to hammer home—and stand my ground on—is that pollsters are incentivized to make their data seem more important, impactful, and meaningful than it is. I always say this, and I say it in the book:

“Data is under no obligation to be meaningful to you in the way you want it to be.”

Rosner: This ties back to your point about seeing unusual numbers from pollsters. When you see numbers that don’t match other polls, it can indicate that the pollster has integrity and isn’t massaging the data. You call it ‘herding,’ where everyone sticks together because numbers closer to the average are more believable to the public than outliers.

Allen:Yes, it’s a “cover your ass” technique because pollsters know how they’ll be judged for accuracy. However, the current measurements for accuracy need to be more scientific and measure the poll’s accuracy.

The current measurements—without going on a rant—are invalid. They don’t measure what they claim to measure. But pollsters know how they’ll be judged, and the mentality is: It’s better to be wrong with everyone than to be the lone outlier. If your poll shows something very different from other polls, you have two options:

1. Don’t release the poll.
2. Fudge the data just enough so your numbers don’t look too different.

This way, if the election result aligns with the consensus, everyone can claim they were “in the ballpark.” But if the outcome deviates from the polls, it’s not just one pollster’s fault—everyone was wrong. This creates a dangerous environment for the independence of poll data because pollsters are judged based on flawed standards and don’t want to stand out.

Suppose we judged polls from a more scientific perspective. In that case, we’d encourage pollsters to use different methodologies, apply different weighting techniques, and be transparent with their data. Whatever numbers they get, they should release them.

Allen: The problem is that if everyone is doing things the same way, if everyone feels pressured to conform to flawed standards, then having 20 pollsters—or even more, as we have now—becomes less valuable statistically. Having 2 or 3 independent pollsters who aren’t herding their data would be more valuable.

Rosner: That’s one of the key takeaways from your book and a message of common sense: Don’t freak out about polls, especially individual polls. There are so many sources of error and misinterpretation in polls that the main message should be: don’t freak out. Don’t waste your time freaking out. Instead, focus on getting people to vote. Polls can help guide where to focus your efforts, indicating which states might be competitive, but you still have to do the work to turn those gettable states into wins.

 In Chapter 4, you discuss ideal polls and present a chart demonstrating how even an ideal poll will show fluctuations. 

Allen: The polling instrument is inherently noisy, even if we know the population with 100% certainty. A poll showing 47%, followed by one showing 52%, doesn’t necessarily indicate movement or a trend. Sometimes, it’s just noise. Yes, rule number one of polls: fluctuation is normal. Individual polls, while important, are just tiny pieces of data in a much bigger picture. 

Rosner: Now, I’ve got one more gripe. You talk about Nate Silver and 538. Nate Silver doesn’t work for 538 anymore. 

Allen: ABC News bought 538 from Nate Silver maybe two years ago. Now, Nate is doing his own thing, working independently with his model.

Rosner: 538 still uses Nate Silver’s model, but it’s no longer tied to him. The 538 team has its methods now. (Important note – I believe 538 now uses a model Morris has brought, and Silver uses the one he previously used at 538 – if you’d like to update)

Rosner: 538’s recipe currently shows Harris up by 2.7%. But if you look at the 20 most recent polls, the average shows her up by 4.5%. Something about that recipe doesn’t add up.

Jacobsen: So, individual psychological factors are also at play here, particularly regarding how people interpret polls, statistics, and public education on these topics. Are we talking about cognitive closure? People want certainty in a context of uncertainty. This need for cognitive closure pushes people to seek definitive answers, even when the situation doesn’t warrant it. People want closure and certainty, even though the nature of polls and predictions is inherently uncertain. I understand from a psychological perspective—why people want certainty. But in science, and again, I would understand if the public and even the media had trouble grasping this concept. 

Allen: The real problem—and the reason I wrote the book—is that experts, not just a few here and there, but a consensus of experts in the field, are misinforming the public. This isn’t just happening in articles; it’s in academic journals and books. Things written by experts for experts wrong.

Allen: They are objectively wrong. I have a list of quotes I share frequently because they’re so easy to interpret, and there’s no context in which these statements make sense. For example, when Nate Silver says, “The poll averages underestimated this candidate by 8 points in the election result,” that’s incorrect. Poll averages do not predict election results, nor do polls themselves. When G. Elliott Morris says, “The polls predicted this candidate would win by 1,” that’s also incorrect. 

When the American Association of Public Opinion Research (AAPOR)—a board of experts hired to analyze poll data—says, “The polls predicted that Hillary Clinton would win,” that is objectively false. There is no context where that statement is accurate. I approach this cognitive dissonance with people, and this is where I gain and lose followers on social media. But my goal is not to gain followers—I want to educate people. The cognitive dissonance is strong because people can’t believe that experts don’t understand this concept.

When you tell people that polls aren’t predictions, most agree, but there’s a reluctance to confront specific experts on this issue. It’s like people are afraid of confrontation. When I call out experts like Tom Bonier or G. Elliott Morris—who frequently post about polls—or even Nate Silver when they say things like, “This poll predicted the candidate would win by 3,” I don’t get direct responses anymore. But they are wrong when they make these claims. It’s easy to say “polls are not predictions,” but it’s difficult for people to accept that experts are wrong. The experts say polls, if accurate,should predict who will win and by how much, but that’s not true.

That’s the psychological factor you’re talking about – cognitive closure. How accurate were the polls? An easy, wrong methodology is currently accepted, and I want to bring scientific standards to the field, nothing more. Well, I also want experts to issue mass retractions for their false claims about what they believe “polls predicted” but that’s another topic.

In my book, I break  down how polls work – and how they should be evaluated -into two parts. The first half of my book is a baseline of education. I build up certain concepts step by step. The second half focuses specifically on political polls. Using the foundation from the first half, I analyze political polls. The content is presented in a way that’s easy to digest and, I hope, somewhat entertaining. It’s simple but explains each concept individually, allowing readers to understand how political polls work.

An ideal poll is one in which the only source of error is the margin of error itself. This is a concept that currently only exists in my book. There is no framework for defining an ideal poll. Analysts in the U.K. and U.S. approach poll analysis with different assumptions. I argue that, although we know political polls are not ideal, understanding non-ideal polls requires knowing what an ideal poll would measure—just like how we teach ideal gases in chemistry or ideal lenses in physics. It’s a theoretical framework to help people understand the basics before diving into real-world complexities.

I didn’t invent the concept of an ideal poll, I just outlined it. Still, it’s foundational to understanding how polls work, especially when analyzing non-ideal, real-world polling data. The concept is rooted in the math. This math has existed for at least 300 years. Still, by naming it and giving it a formal definition, we provide a framework that makes it easier for people to understand. When I say that the book’s first half is an introduction to polls, many experts and smart people who’ve taken statistics classes might think, “I don’t need that.” But I’m telling them, “Yes, you do.” Why? Because they still think a poll predicts the election outcome—and it doesn’t.

These fundamental concepts are what I’m building on. It’s not just about stating facts; it’s about understanding them. Anyone can regurgitate facts—”A squared plus B squared equals C squared” or “P.V. equals nRT”—but spitting out facts isn’t the same as comprehending them. 

Applying the Pythagorean Theorem to an isosceles triangle, to use a simpler example, would not be valid. Even if you don’t know, or remember, that lesson from school – me pointing out “hey, this is wrong, you can’t do that” should be sufficient for the average person to understand who’s right and who’s wrong. The same exact thing is true for the formulas being used to compare polls to elections.The ability to regurgitate some formula is not useful if you don’t know what the output means, or when to use it.

Rosner: Beyond people not understanding these basics, there are so many other abuses of statistical data that it becomes a whole mess. 

Allen: Yes, people don’t understand what polls should do, but they misuse the information in various ways. One of my reviewers, who now works for an NFL team and has a Ph.D. in applied mathematics, said something profound. He noted that statistical literacy is arguably as important as regular literacy.  

Rosner: But we don’t teach statistics. After algebra and geometry, we push students into calculus when most people should learn statistics instead. 

Allen: Unless you’re going into a field like engineering, where calculus is more relevant, statistics should be prioritized. Statistics is almost like logic, but we treat it as just another branch of math. 

Rosner: The concepts aren’t that tough—you could teach nearly everyone the basics of statistics, but we don’t. Yes, and towards the end of the book, you talk about the 2016 Hillary Clinton election. The 2024 numbers look similar to the 2016 numbers. Am I wrong?

Allen: No, they don’t look similar at all. But point out what you think is a similarity, and I’ll tell you where you’re off.

Rosner: 2016, third-party parties, such as Gary Johnson and Jill Stein, had a significant presence.  

Allen: Having third-party candidates affects the data in ways that should be obvious to anyone with a basic understanding of statistics. However, analysts often focus only on the spread between the two major candidates. For example, they might say, “Hillary Clinton is up 4 points with 44% to Trump’s 40%.” Still, they ignore the 5% going to third-party candidates and the 11% undecided. That’s mathematically different from an election where Kamala Harris has 50% and Trump has 46%.

Comparing these numbers as if they’re apples to apples (because both are “up by 4” shows a need for more understanding of polling data. I always say this on social media and get pushback, but I don’t care—because it’s true. The presence of third-party candidates changes the data dynamics in ways that analysts often overlook.

I’ll keep saying this until people fix their nonsensical analysis. No one who truly understands how poll data works would compare those elections apples to apples. I said it in 2020 when Joe Biden was up by 4 in the poll averages in various states. People freaked out like they did when Hillary Clinton was up by 4. However, looking at the data this way needs to be corrected.

In Chapter 24 of my book, I wrote a bit tongue-in-cheek to make a clear point: in an election where the most votes win, 50% plus one vote is all you need. That’s the threshold. That’s the only number we know with certainty. Applying that simple, obvious fact, I prove in the book that experts sometimes need help understanding this.

For example, they’ll compare a poll that says 44% for one candidate and 40% for another with one that says 52% to 48%, as if both are the same because the margin is 4 points. But no—52% is polling across the finish line because we know 50% wins the election. If a poll underestimates a candidate at 52%, they’re still at 51 or 50—they still win. But it’s not even close to the same as if a poll underestimates a candidate at 44%, 43%, or 42%.

That’s a crucial distinction. The numbers the poll gives us—like 44% for one candidate, plus or minus the margin of error—that’s how it should be reported. And if you have 50%, plus or minus the margin of error, that’s the critical number. But people fixate on spread analysis—just looking at the gap between the two candidates—and that’s not how polls should be interpreted.

Chapter 9 of the book discusses this fallacy—spread analysis. People think the gap between two candidates is the only number that matters, but it’s not. No one who understands polling would think that way.

Rosner: Right now, if you look at swing states, you might see numbers like 48–46. In some cases, Trump is at the top, while Harris is at the top in others. But when you add in third-party candidates like Stein or the undecided voters, you’ve got to account for all that. It’s not just about hitting 50% but accounting for the third-party votes and undecided voters.

Allen: The math here is straightforward: if there are only two candidates, as is often the case in the U.S., I’m not calculating the probability that Candidate A wins by a certain margin. I’m calculating the probability that Candidate A gets at least 50% of the vote. In 2020, that’s where things got interesting.

In 2020, there were multiple moments where I realized that I did understand this better than many of the experts. Take Maine or New Hampshire, for example. Joe Biden was ahead in the polling averages by 53% to 40% in both states. FiveThirtyEight  gave him around a 90% chance of winning. But the real issue wasn’t just the spread—it was about whether Biden would cross the 50% threshold and understanding that made all the difference.

Now, when I calculated the probability that a candidate gets at least 50% of the vote when their polling average is 53% or higher, given that there were still 6% undecided, I found it to be over 99%. Even if my calculations were off by a huge factor , that still leaves a probability of 96%, 97%, or 98%. So when I saw these 90% probabilities being thrown around for Biden, I thought, “No, they’re using flawed spread analysis.” They said the spread between Biden and Trump was such that Trump could still overtake him if the spread were off by 10 points. There’s an interesting note in my book that points to the possibility that a very simple clerical error contributed to this probability problem – and no one there caught it!

But when you understand how poll error calculations work, you realize that the probability of a candidate outperforming their poll number is much higher when there are 15% undecided voters than when there are only 5%. This sounds obvious when I say it. If there are more undecided voters, the final result is more uncertain. But spread analysis doesn’t account for that uncertainty. It treats a 42-40 poll as if it’s the same as a 50-48 poll, which is fundamentally incorrect.

Rosner: Let me throw some numbers at you. You’re giving Harris close to a two-thirds chance of winning right now. In 2016, Hillary was up by about 5% in the national aggregate, but that’s not helpful because of swing states and the Electoral College. Then Comey dropped the FBI investigation news with 11 days to go. She was up 5% but won the popular vote by only 2%. That announcement may have cost her 1% or 1.5%, but nobody knows.

In 2020, Biden was up by 8% to 10% in the week before the election but won by 4.5%. Now you’re saying Harris has about a 2-to-3 chance of winning, but what’s happening?

Allen: The analysis done by experts and the media in 2016 goes like this: Hillary was up by 5%, then she lost by 1%. Therefore, the polls were off by 6%. But this is not a valid analysis, and here’s why. Hillary Clinton’s polling average wasn’t above 47% in any swing state—none—not in Michigan, Pennsylvania, Wisconsin, or even Maine. So that means there were a lot of undecided voters still on the table.

Rosner: Looking at 47% for Clinton and around 43% for Trump leaves 10% of voters who hadn’t decided or considered third-party candidates; that undecided group could heavily influence the election’s outcome.

Allen: Yes. I have the exact numbers in the book, and they vary by state—46-41, 45-42—but the key point is that in no swing state was Hillary’s polling average above 47%. This meant a significant portion of the electorate was still undecided, and that’s where the real uncertainty lay.

Oh, here it is, right in front of me—Pennsylvania. Hillary Clinton’s polling average in Pennsylvania was 46.3%, and Trump’s was 43.9%. So, the analysts said she was up by about 2.5%. Now, what are they doing today? They say, “Oh, Kamala Harris is only up by 2.5%, and Hillary lost, so there’s this normal polling error.”

No, that’s bullshit. We’re calculating the probability that Kamala Harris will get at least 49.5% of the vote because third parties and fringe candidates will likely take about 1-2%. Even if we simplify the math and say the probability she gets 50%, that’s what we’re analyzing. There’s no magic number in polling except for 50%. And it’s not magic—it’s math. You get to 50%, and you win, period—end of story. 50% is the number that matters. If you get 50%, you win. That’s it.

So let’s say Kamala Harris’s polling average is 49%, and Trump is 47%. I’m not analyzing that poll by saying, “Harris is up by 2.” I’m analyzing it by asking the probability that she reaches 50%. In my forecast, I account for third parties taking about 1.5% of the vote. So, the real analysis is the probability that she gets to something like 49.3%, which is enough to win.

Rosner: So, to simplify: in 2016, Hillary and Trump’s combined polling numbers increased to less than 90%. The remaining undecided or third-party votes, which made up 10-12%, made all the difference. However, in this election, Harris was 48%, and Trump was at 46%, adding up to 94%. That means only 6% are left unallocated. It’s much easier for Harris to make up the 2% to get to 50% than for Hillary to bridge that gap in 2016.

Allen: Hillary had a much steeper climb to get to 50% or even 49% because there were more undecided and third-party votes in play. In this election, the smaller number of undecided voters makes it easier for Harris to reach 50%. 

I’ve taken a few notes and want to explain why I wrote the book. The simplest reason? Because experts are misinforming the public. This isn’t a rare or one-off issue. Experts analyze polls by the spread, margin, whatever you want to call it: the difference between the top two candidates. But that’s an internally invalid metric. Spread doesn’t measure what they claim it does. Spread analysis needs to capture the full picture.

In the book, I explain why this is the case. An ideal poll is a poll where the only source of error is the margin of error. It’s possible to have an ideal poll, but political polls aren’t ideal—and that doesn’t matter. The math behind polling, from which we get the margin of error, is the same math that underlies an ideal poll.

Rule number 1 of polling data: fluctuation is normal and expected. Individual polls should be taken with a grain of salt. The numbers will go up and down. If a pollster consistently releases the same numbers—49, 49, 49, 49—I’m highly skeptical of that pollster because it’s statistically impossible to get the same number consistently if you’re conducting polls correctly.

So even in an ideal poll, you wouldn’t expect the same number every time. Even in an ideal poll, there should be fluctuation. If you’re getting the same number repeatedly, something is off. In non-ideal polls, we should expect even more fluctuations. 

What I would say about the last part of the book is that there are some important statistics. The book’s first half is about the why—the foundation of polling and why the public and even experts misunderstand it. The book’s second half is the what—and this is where it blows people’s minds. My analysis is scientifically valid, and it makes predictions.

If my analysis is correct, it should hold that candidates who poll closer to 50% tend to win, regardless of how much they are up by. So, a 50–48 poll is better than a 46–40 poll, even though the margin (+2 vs. +6) is smaller. If my analysis is wrong and spread is a valid way to interpret polls, then the opposite would be true. But it’s not—and that’s the counterintuitive yet obvious conclusion if you think about it.

Rosner: That makes perfect sense. Our minds have been conditioned by spread analysis to the top.

Allen: If this book had been written 100 years ago, it wouldn’t have been controversial. Most people would’ve considered it obvious. But because of this obsession with margin analysis—up by two versus up by 6—sometimes being up by two is better, mathematically, than being up by 6. It’s a mathematical fact. It’s provable and observable; we have data to back it up. I put all of this in the book.

Rosner: It’s like with the Dodgers. Their lead kept shrinking over the last two months of the season. I always asked myself, is it better to be up by eight games in July or up by two games with a week to go?

Allen: Bingo. That’s the perfect question. I get this all the time with political data. People ask me, “Is it better to be up 49–47 in a poll average or 46–40?” I know the answer, but the answer given by experts is different.

Rosner: Who’s right? They can’t both be true.

Allen: Only one of these can be used. This has led to a combative debate in the field. Some experts who used to be friendly with me are no longer because they realize that my work and their work can’t coexist. There’s this dissonance—they cling to how things have always been done. But my work is provable. It’s objectively correct, it’s been tested, and it holds up. To Rick’s point, it’s not hard to understand.

Allen: Most of this can be taught to high school or college students. The fact that experts still get it wrong—still analyze polls by margin, by who’s ahead and by how much—is baffling. This is a new way for most people to think about polling, but it’s the correct way. The old, misinformed spread analysis has confused people for too long. This approach simplifies things and aligns with the reality of how polling works.

There’s a quote in the book: “Even numerate people can be misled when they’re misinformed. ” For many years, people have talked about the spread as if it were the golden standard—the metric we should use. These people have Ph. D.s in statistics, but in practice, even experts are misled by the spread.

Rosner: In gambling, you bet on the spread. But applying that mindset—like you do in NFL betting—to politics becomes deceptive. Spread is a misleading metric. Spread proclaims to measure who is ahead and by how much, but it fails on both counts.

Allen: Absolutely. I talk about this in the book. It’s simple to prove—anyone can do it. You can even use real data to show it. We could cover this in another call.

Jacobsen: What have early sales of the book been like, and how long have you been working on it?

Allen: Sure! So, presales opened on September 2nd. I aggressively promoted on social media and other channels, and sales did well in the first few days. After that, I took a short break to set up some media appearances. The book was officially released on September 23rd when preorders were shipped, and regular orders opened. I did another push then, and sales spiked again. It’s been a peak-and-valley situation since.

But realistically, my mentality is that I have about a month and a half until the election to capture people’s attention. When I started, it was two months. The truth is, after the election, I expect more interest from academics. I’ve already had invitations to speak at universities after the election, but that’s more of a niche market. My book isn’t just for academics or people in the field—it’s for anyone who wants to understand polls better. The average person tunes into polls only in the months leading up to elections, so that’s where my focus has been.

Rosner: Do you work with the Florida Elections Project or the early voter guy?

Allen: No, I haven’t worked with him. I’ve been focused on my projects for now. I follow all these people because they provide interesting insights. I always tell people you must take the good with the bad with these analysts. Whenever I criticize Nate Silver, Tom Bonier, G. Elliott Morris, or whoever, people assume two things: First, they take it personally, which isn’t the case—I don’t know them personally. And second, they assume I’m saying they don’t do any good work. That’s not true at all.

I always say, “They’re right about this. They do good work here.” I can learn from people, and many analysts do better work than me in other areas, like early voting counts or election day calls. For example, Dave Wasserman is great at calling elections when the votes start coming in. I’ve tried to do what these people do and couldn’t improve on it. So, I follow them and learn from their expertise. But in cases when they do poor work, I criticize that and say, “No, that’s not right. You need to learn a bit from me.”

As someone who never stops learning, I’m even wearing my “Never Stop Learning” shirt today, and I take both sides of the coin. You take the good with the bad. There’s always value in learning from others, even if you don’t agree with everything they do.

The background of this book started in 2016, during that notorious election. After Sam Wang announced his 99.9% probability that Hillary Clinton would win, he famously said he’d eat a bug if she lost. And to his credit, he did eat a bug. But many other forecasters still need to follow through on their grand promises. They said they’d delete their accounts if they were wrong about some things in 2020, and again in 2022,but they have yet to do so.

The book’s origins go back to 2016 when I saw people with big reputations misinforming the public. Before I got into the margin and spread analysis, I thought, “How can so many smart people not understand that states are correlated?” What happens in Wisconsin affects Michigan, which affects Pennsylvania.

Rosner: So, what was your first step into forecasting?

Allen: My first attempt at building a forecast was in 2008. I was a freshman in college with my laptop open, trying to calculate poll averages. I thought I had it all figured out. Then, I realized that Ohio is correlated with Michigan, which is correlated with Indiana and Iowa. I didn’t know how to do that math back then, so I shut my laptop, went to the gym, and didn’t think about it for another eight years.

By 2016, I had learned more and improved my math skills. I built a forecast that gave Hillary Clinton a 70% chance to win—not because she was up by six or anything like that, but because I saw she was only polling at 46% or 47% in the swing states. Trump still had a path to victory. I realized that if he won Wisconsin, there was a good chance that he would also win Michigan and Pennsylvania. These states are correlated, and all the forecasters who put Hillary at 99% to win didn’t account for this.

It’s obvious, but I understand it might not be for the average person. However, it should be obvious to any statistician or model builder that what happens in one state is not independent of another.

Rosner: So, did Nate Silver think the same about correlated states back then?

Allen: Nate Silver acknowledged the correlation between states and calculated them very well, one of the things I learned from him, but his model gave different probabilities than other forecasters that year. He was more cautious than others who were giving Hillary 99%. He gave her around a 70% chance, similar to my forecast, because he recognized the possibility of Trump winning correlated states like Wisconsin, Michigan, and Pennsylvania. Nate’s thinking was closer to mine, but many others completely overlooked that factor.

Yes, At that time, Nate Silver was the only mainstream forecaster who got the numbers right to understand that states are correlated. Huffington Post, Sam Wang, and several others said, “Well, if she loses Wisconsin, that’s fine because she can still win Michigan, and she’s 90% to win Michigan. Even if she doesn’t win Michigan, she can still win Pennsylvania, and she’s 86% to win Pennsylvania.” Or whatever the numbers were. But the reality is, as soon as one of those dominoes falls, all the downstream probabilities drop dramatically—from 90 to 40, from 86 to 32—because those states are correlated.

To summarize, I wrote the book backward. I knew what I knew but only fully understood why later. From 2017 to 2020, I was reaching out to experts, academics, and people in the field, saying, “Look, there’s something in my research that shows how much someone is “up” in a poll— is not as important as their actual poll number.” I knew I was onto something but couldn’t fully wrap my head around it. I asked if anyone wanted to take it from there. No one was interested, so I had to do it alone.

From there, I started working backward. I gathered all the data from 2004, through, 2018, at that time. I knew what I was trying to prove, but in statistical literacy, you can’t just say, “Here’s the formula; deal with it.” You have to prove it—you have to show your work. So, I worked backward, asking, “How do I know this?” Chapter 16: compensating error. “How do I know that?” Chapter 11: weighted results. “How do I know that?” Chapter 7: present polls versus plan polls and a simultaneous census. And, of course, Chapter 4: ideal polls. Chapter 2: the margin of error for polls in very small populations, where a census is easily conducted.

While the book was written backward from my perspective, it’s logical and straightforward for the reader because it builds a foundation. You need to understand one concept before moving on to the next.

Rosner: That makes sense. Yesterday, after you told me, I explained to my wife how Harris’s 48-46 in some states is much better than Hillary’s 44-40 in 2016. But I only had a limited “math window” with her before she said, “I don’t care.” I need to get it across in time!

Allen: Yes, it takes time to grasp. But to understand this election—and my book—your point is exactly right: a small lead close to 49 or 50% is better than a larger lead farther from it. It’s counterintuitive because we’ve been trained to think about margins, but it’s  mathematically and logically true. The closer you are to 50%, the less room for things to go wrong between now and the finish line.

Rosner: That does make sense if you let go of the spread mentality. Analysts and academic articles use the same language: “She’s up by 2” and “He’s up by 4.” I had someone jokingly send me an article today from The New York Times or maybe their website using that same spread logic.

Allen: Right, that’s the spread mentality I’m trying to break down. It’s ingrained in how people think about elections. Still, as you understand polling better, you realize how flawed that thinking is. The closer a candidate is to 50%, the more likely they will win, regardless of the margin.

If people say things like, “What if the polls are off by as much as they were in 2016, 2020, or 2022?”—you can’t make those comparisons. Those aren’t apples-to-apples situations. First, comparing midterms to a general election is a bad comparison. Even comparing 2020 to 2024 is difficult because the variables are different. And 2016 is an outlier. You’re taking an outlier and trying to apply it to 2024, which is not a sound method.

It’s like in a movie when two people are on the floor, both trying to reach for a gun. It’s better to be 6 inches away from the gun. At the same time, the other person is a foot away rather than 3 feet away, while the other person is 5 feet away. The analogy I use in the book is a footrace. Imagine watching a race and knowing one runner is ahead by 2 meters. Now, is that the most valuable piece of information? It depends. If the finish line is at 50 meters, knowing how far someone is from the finish line gives you a lot more information than just knowing who’s ahead by 2 meters.

So, let’s say you’re in a race, and you’re ahead by 2 meters. Would you rather be ahead 49–47 or 42–40? You’d rather be closer to the finish line, right? That’s where I start. It’s better to be closer to 50 for an equal spread percentage. That makes sense.

Then I ask, “Would you rather be ahead 49–47 or 44–40?” At 49%, you only need one more percent to get to 50. At 44%, you still need 6. So, the probability that the person behind you overtakes you is greater when you’re farther from 50%. When people focus on the spread in elections, it’s almost like football fans worrying about covering it. But in an election, there’s no significant benefit to winning by 10 points instead of 2.

Rosner: Right, the goal is just to get past 50%.

Allen: When calculating win probability, we ask, “What’s the probability that this candidate gets at least 50% of the vote?” I’m not trying to determine whether a candidate wins by 10 points because that doesn’t matter. Yeah, I can calculate those probabilities, and yeah, the probability of winning “by 10” is higher at 44-40 – but the average person doesn’t care about that. They want win probability. And in our elections, there’s no added benefit to winning by 10. Math and logic people like to hedge with “all else equal” but the truth is, in elections, there are always so many variables.

Rosner: Like in 2016 when James Comey threw a bag of dog shit on the track with his last-minute FBI announcement.

Allen: Hillary was leading, and then Comey threw dog shit on the track with 11 days to go. But in 2024, things are more locked down because fewer undecided voters exist. So, according to my logic, there’s less chance for something like that to throw things off.

Harris is likelier to make it through, especially when fewer undecided voters  left, with some data supporting they’ll lean Democrat, very different from 2016In 2016, Hillary lost a significant portion of those undecideds who flipped to Trump, which cost her a huge chunk of her perceived “lead”. Some of her voters probably didn’t even show up to vote because of the perception that she “had it in the bag,” which is another issue with how the spread is often misinterpreted.

If I tell you, “She’s up by 6, she’s got this,” it can lead to voter complacency. This is where Nick Panagakis comes in. As far as I can see, he’s the only historical researcher who identified this issue. Political polling has existed for over 100 years, since Gallup and Literary Digest. I discovered his work in old journal articles and newspapers. Still, no one knows his name anymore—his work has essentially been lost to history.

Rosner: Panagakis sounds like someone ahead of his time.

Allen: He’s the only person I’ve found who corroborates my findings. He published in a few academic journals, but his work needs to be noticed. In the 1980s, he came up with some essential rules of analysis. He said, “Rules of analysis are necessary—not as simple as ‘an 8-point lead is safe, and a 2-point lead is close.'” That sounds eerily like what I’m saying today.

Rosner: That sounds like what you’ve been arguing about.

Allen: The eerie part is that I didn’t come across his work until 2021 or 2022. I was looking for past research similar to what I was saying, and then I found this guy. He’s got his chapter in my book, Chapter 21. When I found his work, I was floored—he was saying many of the same things I’m saying today, and he had the evidence to back it up.

Panagakis argued that undecided voters sometimes split unevenly. Often, they go disproportionately to the candidate who’s behind. If you account for that, polls that appear wrong are very accurate. His work would have been criticized, modified, and accepted in a proper scientific field. But because political polling is so contaminated with this obsession over spread logic—who’s up and by how much—his work was buried.

Rosner: That’s fascinating. 

Rosner: Unfortunately, this spread mentality has clouded the field for so long. Statistics as a field has its shameful history, with a lot of it developed by racists who used population statistics to push agendas—proving white superiority over non-whites. It’s a terrible history if you dig into it.

But to shift the focus for a second, I’d like to talk about the history of presidential polling and get your thoughts on something. Presidential polling began around the end of FDR’s era in the 1940s, about 80 years ago. Suppose you look at the history of presidential approval. In that case, it has steadily declined over time, with one major exception—9/11, which gave George W. Bush a huge bump in approval as the country rallied behind him for a few months. But aside from that, approval has been declining from president to president, regardless of whether they’re good or bad. As we get more polarized, average approval goes down. Do you have any thoughts on this?

Jacobsen: It makes sense that approval ratings would decline as the country becomes more polarized. We’re in a time where fewer people are willing to give the benefit of the doubt to a president from the opposing party. The rise of media echo chambers and the constant stream of information also make it easier for people to entrench themselves in their views, which means fewer opportunities for a president to win over the other side.

Allen: That’s true. We’re increasingly seeing a political climate where people feel they’re choosing the lesser of two evils. It’s not that they strongly support one side but that they feel the other side is worse. And regarding Donald Trump, MAGA is almost an exception in recent political history. If you look at Mitt Romney, John McCain, or George W. Bush, there wasn’t the same personal loyalty to the candidate. However, with Trump, a large number of his supporters turned out to vote specifically for him. You didn’t see that kind of enthusiasm for Joe Biden, and you likely won’t see it for Kamala Harris.

Rosner: Barack Obama might be an exception on the Democratic side, however. His following wasn’t so much a cult of personality but more about empowerment and inspiration. But overall, the polarization of U.S. politics is fascinating, especially compared to countries with five or more political parties. In those places, voters can more easily shift from one party to another. If they don’t support one party in a particular election, they ideologically move to the next one.

Allen: But in the U.S., moving from Democrat to Republican or Republican to Democrat is a huge shift. It’s a big ask to go from voting for Donald Trump to voting for Kamala Harris or vice versa. That’s a major ideological jump. So, the data we get regarding popularity in the U.S. can be skewed. Take Mitch McConnell, for example. He regularly has around 30% approval in his home state of Kentucky. However, he still won reelection because, to Kentucky voters, he’s the lesser of two evils. Whoever the Democrats run is always portrayed as far-left, out of touch with Kentucky values, and so on.

Rosner: That’s a pattern we’re seeing more and more—this hyper-polarized environment. Historically, it is hard to look at approval ratings because this extreme polarization has only intensified over the last 10 to 20 years. Having a charismatic candidate greatly helps, but it has been a while since we’ve had one. The last truly charismatic presidential candidate was Obama in 2012. In 2016, both candidates—Trump and Clinton—had high negatives—the same thing happened in 2020. Now, however, Harris has some charisma. She has amazing hair, historically a big deal in elections. JFK, great hair. Clinton has pretty good hair. Reagan has amazing hair.

Allen: There’s something to appearance in politics. It’s not a deciding factor, but it plays a role. A psychologist could speak more to that than I can, but there’s something about how a candidate looks and sounds. Someone who looks the part and is eloquent can have more appeal than someone who might be smart and have good policies but doesn’t come across as well visually or rhetorically. Rosner: There’s something to that. Presidents are like America’s flight attendants. Remember how exciting flight attendants were in the 1960s? They wore mini skirts and were the subject of many, many fantasies. Over the years, they’ve been replaced with the idea of flight attendants who don’t have to be sexy. But presidents are, to some extent, America’s sexy cheerleaders, and it’s good to want one who’s “cute.”

there’s an element of that in how we perceive political figures.

Allen: But I want to get back to something I mentioned earlier. I was flipping through my book, and a point I repeatedly make is crucial on social media and in the book. Regurgitating a fact is not the same as understanding it. I’ll elaborate. Analysts and experts often regurgitate certain facts like, “Polls are snapshots.” That’s correct, but they must truly understand what that means when analyzing the data.

Rosner: They say one thing and then contradict themselves in the analysis.

Allen: They’ll say, “Polls are snapshots,” but then they analyze them as predictions. There’s a disconnect. Saying you understand something and demonstrating it are two different things. This is why I was so excited when the publisher asked me to write a book on statistical literacy. Instead of writing for experts in a bubble—which they need to get out of—I also had the chance to explain these concepts more broadly.

One key concept I talk about is the idea of a simultaneous census. What does a poll measure? That seems like a dreadfully simple question. But if you ask experts, you’ll get various answers, and many will repeat a textbook definition, calling it a “snapshot.” Then, when you ask them to explain what the poll means, they often need to be more knowledgeable. Their words prove they don’t need to learn what the data signifies. They’ll say, “The polls predicted…” No. Polls are not predictions.

Rosner: So, what do polls measure, in your view?

Allen: Polls are an estimate of a simultaneous census—a snapshot of a candidate’s base of support at that moment in time, not a prediction of the future. That’s why the margin of error plays such a huge role. Take a candidate polling at 49%. Before we even discuss undecided or people potentially changing their minds, that 49% could be 50% or 51%. Or it could be 48% or 47%.

Mathematically speaking, 49% is just as likely to be 50 or 51 as it is to be 48 or 47. It’s more likely to be close to 49, so candidates polling at 49% do well. Now, compare that to a candidate polling at 44%. That 44% could be 45% or 46%, but it could also be 43% or 42%. Even at 46%—the high end of their base of support—they’re still far from 50%, which means they can easily be overtaken.

Rosner: So, it’s not just about being ahead, but about how close you are to 50%.

Allen: The closer you reach 50%, the better your chances. That’s why analyzing polling numbers properly is so important. Candidates polling at 44% are in a much riskier position because, even at their best, they’re still far from the finish line.

Allen: When I talk about the simultaneous census concept, the question it answers is: What portion of the population currently supports this candidate? It’s about understanding what a poll is measuring. In Chapter 8, I explain this with a real experiment. Suppose you had asked everyone in the population the same question simultaneously. In that case, the result would fall within the margin of error.

Rosner: You also talk about ideal polls.

Allen: But let’s talk about bad snapshots—like putting a filter on your phone to look super hot.

Rosner: Right, like Rasmussen.

Allen: Yes. Rasmussen got kicked out of the 538 aggregate for being too biased.

Rosner: So, what about those “bad pictures” of the population?

Allen: The reason Rasmussen was kicked out of the 538 aggregate wasn’t necessarily because they were too biased—it was because they didn’t share their methodology. When 538 asked, “You’ve got these numbers, but where did you get them from?” and Rasmussen essentially said, “Don’t worry about it,” that’s a red flag. It doesn’t matter if the data is legitimate—if someone says, “I’ve got these numbers,” then refuses to explain how they got them, that’s problematic.

It’s simple: your data should only be included if you’re forthright about how you conducted your polling. This is true in any field of science. It automatically loses credibility if you’re clear about your methods and your data can’t be replicated. Nate Silver has a slightly different opinion. He acknowledges that there might be a good reason not to include Rasmussen because of their methodology. Still, he argues that Rasmussen’s overall accuracy wasn’t bad, so it’s debatable.

Rosner: So, I could just put out a “Carl Allen Poll,” saying Harris is at 48 and Trump at 46, and when someone asks, “Where did you get those numbers?” I could say, “Don’t worry about it,” that would fly.

Allen: Your data shouldn’t be considered if you’re not doing real research. Accuracy alone isn’t enough—transparency is key. Without it, the data is useless.

Rosner: What about methodologies that bug you? When I look up how some polling companies operate, I see things that bother me. For example, some companies use a paid panel—they recruit 5,000 people, try to make them demographically balanced, and then ask a random sample of 1,000 from that panel each week what they think. They pay them a little, but what if the pool is contaminated? What if someone is peeing into the pool?

Allen: Yes, that’s a concern.

Rosner: And then there are companies still using landlines! I’m old—I still have a landline—but that’s outdated. What do you think of those methodologies?

Allen: I find methodologies suspect, but there’s a big “but” here. It’s important. The issue isn’t necessarily the method itself—it’s how you use it. For example, using a paid panel can introduce bias. However, you can still get valuable information if the data is weighted correctly and the methodology is transparent. It’s the same with landlines. It seems outdated, but combined with other methods, it can still contribute to a representative sample. The key is transparency and understanding the limitations of each method.

Allen: The purpose of a transparent methodology is to ensure that it is conducted with scientific goals in mind. The goal of transparent methodology is not to confine everyone to a strict framework and dictate that everything must be done in a specific way. Because it is so imprecise, polling data is a science—but an inexact science. Suppose someone develops a technique to achieve a better random sample. In that case, we should not reject it simply because it does not align with established guidelines.

I strongly advocate for diversity and innovation in methodology, provided there is transparency about how it is being conducted. Whether it’s landline polls, cell phone polls, online panels, or mixed methods, I support them all. As an analyst who examines this data, I want to determine which methodology is effective and which is not and how a flawed methodology could be improved.

The book discusses a significant point about Literary Digest, which became infamous for conducting political polls before elections. Their methodology involved sending out mail-in surveys to their subscribers, which produced a vast sample but not a random one. Their subscribers were typically more affluent and urban, which skewed the results. Despite this, for several elections in the 1920s and early 1930s, their unscientific polls produced results so close to the actual outcomes that even scientists concluded the results were reliable.

However, in 1936, Literary Digest “prediction” as it was reported, said  that Republican candidate Alf Landon would defeat Franklin D. Roosevelt with 57% of the vote. When the election results came in, Roosevelt won in a landslide, receiving 62% of the vote. This massive polling error exposed the flaws in their methodology—such as the failure to account for the demographic biases of their mailing list. In hindsight, and only in hindsight, did it become clear that their methods were unsound, but this significant failure revealed those flaws. The perception of being “accurate” by an unscientific measure gave them credibility they didn’t deserve. This is the exact mindset people still have today. Note how casually the experts that demean the Literary Digest judge a poll’s accuracy by how well it predicted past elections. Ask them what rating they’d have given to the Literary Digest prior to 1936. They’ll block you for it. A similar issue occurred in 1948 with the famously incorrect “Dewey Defeats Truman” headline, where polling missteps, and misinterpretation,also contributed to the erroneous forecast.

Allen: In Chapter 5, I emphasize that fluctuation in poll results is normal. Even the best pollsters sometimes produce slightly inaccurate results—that’s how it works. We cannot view polls as instruments that are supposed to be perfect. More independent pollsters—independent being the keyword—who do not skew their data would significantly advance the field. This will only be possible if better methodologies, like the ones I propose in the book for analyzing poll accuracy, are adopted.

Rosner: So, you should take the opportunity to plug your book hard here.

Allen: Sure. The Polls Weren’t Wrong will change how polling data is analyzed and understood in the U.S. and worldwide. I make this claim because the book’s scientific approach is one that will win out over the current methods.All the analyses and methods I use are grounded in science. The methods for analyzing pollscurrently  in the U.S. and globally are not scientifically sound. Polling will improve with a greater understanding of history and the adoption of better scientific methods. Whether this shift happens in two or twenty years isn’t for me to decide, but the world would undoubtedly benefit from it. People would have a clearer understanding of what poll data means.

Last updated May 3, 2025. These terms govern all In Sight Publishing content—past, present, and future—and supersede any prior notices. In Sight Publishing by Scott Douglas Jacobsen is licensed under a Creative Commons BY‑NC‑ND 4.0; © In Sight Publishing by Scott Douglas Jacobsen 2012–Present. All trademarks, performances, databases & branding are owned by their rights holders; no use without permission. Unauthorized copying, modification, framing or public communication is prohibited. External links are not endorsed. Cookies & tracking require consent, and data processing complies with PIPEDA & GDPR; no data from children < 13 (COPPA). Content meets WCAG 2.1 AA under the Accessible Canada Act & is preserved in open archival formats with backups. Excerpts & links require full credit & hyperlink; limited quoting under fair-dealing & fair-use. All content is informational; no liability for errors or omissions: Feedback welcome, and verified errors corrected promptly. For permissions or DMCA notices, email: scott.jacobsen2025@gmail.com. Site use is governed by BC laws; content is “as‑is,” liability limited, users indemnify us; moral, performers’ & database sui generis rights reserved.