US Politics Mega-thread - Page 5885

Alright, I finally have a chance to explain this issue. I'll also take this chance to explain the entire issue of election predictions because that's the context in which we actually care about probabilities and how they are interpreted. There's a lot of reading you could do depending on how much you care about the issue. It's rarely discussed though, because most people don't know and don't care about the "what is a probability" issue and just take the most intuitive approach that happens to be the frequentist one. It's one of those mistakes that is really easy to make because even among the people who are supposed to know better, it's common that they just don't really care.

I'm going to start this discussion by linking this wiki page on interpretations of probability, which contains with it a link to the "Bayesian probability" page previously referenced. This takes a more philosophical than mathematical approach to the issue of probabilities (and philosophical Bayesianism is... kind of odd), but it is pretty comprehensive and you'll find more than you need to know. I'm actually going to just focus on a very limited dichotomy, the one most relevant to mathematical statistics: the frequentist versus the Bayesian-logical approach. Frequentism considers probabilities in terms of limiting frequencies - like if I say I have a 45% chance of rolling a 6, that means that if I rolled the dice 100 billion times then I'd get something really, really close to 45 billion 6's. A Bayesian-logical approach (right now the most commonly used Bayesian approach, and henceforth referred to simply as "Bayesian") interprets probabilities as "degrees of belief." To put it in the context of election predictions: if, from a Bayesian perspective I say that "Hillary has a 65 percent chance of winning the election" what I'm really saying is, "based on the model that I have designed, I am 65 percent sure that the outcome of the coming election is that Hillary will win." This difference is one that manifests itself mostly in how you can use the machinery of probability theory in making predictions about future events.

The frequentist approach is basically the classical approach to probability, and the one that is mostly quite mature as a field of mathematics. I'm not going to go into it much; either you have the relevant mathematical background to already know what "classical probability" is, or you won't get it. The best book on probability theory from a frequentist/classical perspective is William Feller's two-part series An Introduction to Probability Theory and Its Applications (Part 1 Part 2) which is at a graduate level of mathematics, but is also the definitive text on classical probability (and one of my personal favorites among math books). Back when Feller wrote his book (1950 for the first edition), Bayesianism wasn't really very well-received in the mathematical community; it was a "big new idea" that more experienced mathematicians were rightfully skeptical about; it did end up being very useful but the concerns about its questionable reliability were really spot-on.

The Bayesian approach is an interesting outgrowth of the Bayes' rule or Bayes' theorem, a simple and almost-trivial statement of relation between conditional probabilities (e.g. probability of some event happening given that some other event happened). This simple rule forms the backbone of a range of techniques called Bayesian inference which use Bayes' theorem as a tool to update predictions based on new evidence. There are a lot of approaches, and a lot of techniques that are used, and unlike classical probability I wouldn't call Bayesian inference anything close to mature. It's proven to be extremely useful in a wide range of applications of probability, but it does suffer from a few issues, which I'll get to in a bit. There isn't really any definitive book I can recommend on Bayesianism since it's not a mature field, but I will offer Phil Gregory's Bayesian Logical Data Analysis for the Physical Sciences (link) as something that develops a Bayesian-logical approach to probability in a useful way.

Frequentism basically only uses the classical machinery of probabilistic theory, which is useful but limited in what you can accomplish with it. The use of Bayesian approaches expands what probabilities can be used for to a lot of further applications - the one we are interested in in this case is the application to events that happen only once, something that from a philosophical perspective is very irreconcilable with the definition of frequentist probability. In the Bayesian-logical context, we use evidence (here, polls and possibly fundamentals) within some model to get what we hope is an increasingly accurate picture of how sure we are that the evidence points to a certain outcome. Although the validity of Bayesian approaches is very clear by now, the classical criticism of them being easily misled is a very poignant concern that justifies the skepticism of the mathematicians who were suspicious about it.

So if you've taken any statistics or probability coursework, the basic idea of Bayesian inference is rather simple: you have your prior assumption about the state of the world, you have some evidence, and then you have a marginal likelihood (a factor that you use to ensure that the probability of all events sum to one). The basic idea that is used is that you try to make as uninformative an assumption about the past state of events as you can, given only what you automatically assume to be true (not a trivial topic; I'll link maximum entropy as one approach to look at but there's a lot to this), and then you let those assumptions just lose their influence as you add new data. The real issue with Bayesianism, though, is this: when you choose priors and probabilistic representations of the likelihood of your observed data, you prescribe a probabilistic structure to that data. Nothing inherently wrong with that, that's basically the entire idea of what a "model" does - in math, in science, or anywhere else. The problem is that if your model sucks then your results do too, and Bayesianism is especially vulnerable to this issue.

Now that we have established a context for what Bayesianism is, let's get into the issue of polls and predictions. I'm going to focus most on FiveThirtyEight, both because it's popular here on TL and because I personally have always been very impressed with Nate Silver's work on election predictions. I'm also going to briefly touch upon the Princeton Election Consortium predictions, which I'm less fond of but that will make a good example for one point I have to make about models. As with any non-trivial predictions, the models used are both complex and partially opaque (partly to avoid people stealing their stuff, partly because not too many people care about all the technical details). To put it simply though, most models focus on two general broad ranges of factors: poll results, and fundamentals (economics, historical results, approval ratings, etc). If you want more background reading, look at this paper on fundamental models and this Pew piece on polling methodology.

The major idea behind 538's model is this general line of thought: averages of polls are better than single polls, the results of polls are accurate to some measurable margin of error, and fundamental factors have an effect but are insufficient to predict the results. If you want some further reading on the model that Nate Silver and 538 have made, you should look at their list of pollster ratings, Nate Silver's technical explanation of how his pollster ratings work, their article on the usefulness of polls in making predictions, and their Endorsement Primary article (one of their "fundamentals" factors, of many). As an example of a different methodology, the PEC uses an approach called the meta margin as a "buffer factor" to predict elections, and their model gives a much tighter bound on the chances of Hillary Clinton winning.

You may or may not remember that Nate Silver became famous four years ago in the 2012 elections, when he differed from most other predictions and gave a model that looked ridiculously biased in favor of Obama, but ended up being the exact result of the election (he guessed 1 Senate seat and 0 electoral votes wrong). While he admitted that some luck factors played in his favor, the other thing that helped was the soundness of his statistical methods. What I remember most about Nate Silver in 2012 was the controversy over how Gallup polls (the most reliable pollster in terms of raw prediction rate) predicted that Romney would win, and how he had a rather scathing criticism of that assumption that generated a lot of controversy but ultimately ended up being really, really spot-on. That is basically how he became famous; I'm fine with that because I have generally been very impressed with the technical soundness of his statistical analysis (despite some factors I disagree with him on) and I think that the 538 model is the one most representative of the truth at this point. He did have a somewhat famous failure to predict elections in the 2015 UK General Election (read this Quartz piece, this 538 admission of failure, and this follow-up). I attribute that to the fact that his success in 2012 was well-founded in good statistical modeling, but being able to predict results with that uncanny level of accuracy is rare. Polling is a difficult task that is getting increasingly unreliable; read the earlier Pew piece on polling methodology and the 538 piece on polling accuracy if you want more details.

Now, after all that, we can talk about Bayesianism in the context of polling predictions. Nate Silver's approach is very Bayesian and best thought of in the previously described Bayesian-logical framework. His inference comes from a form of the model I have described, in which he updates the forecast based on new polls (both state and national), and fundamental factors (but only in the polls-plus model). Everything he does culminates in a bunch of distributions for the predicted result for each state (likely some variant of a Student t distribution, with a few added details), from which we can predict the results of the electoral college. The effect of the "fundamental" factors is to essentially "tip the scales" in one direction or the other in a way that makes sense; Nate Silver has noted that while the polls-plus models are more accurate than polls-only the majority of the time (57% versus 43%), they are only useful when you look at both of them together. In any case, this sounds very much like the Bayesian-logical framework that I have previously explained about.

Your objection was about that they run ten thousand simulations so you don't see how it could be a "state of belief." What's happening here is that he is applying a technique called a Monte Carlo simulation. The simple explanation is that the probabilistic description is analytically intractable, but it's based on a whole bunch of simple probability distributions which are analytically trivial to analyze. So all you do is you run a simulation, where you generate a completely random sample from each individual distribution, you combine those results, and you record the ultimate result of all of those factors. Then you repeat until you have enough. The methods for proving "what is enough" are too difficult to get into, and I'm not going to bother trying to explain this. But I will tell you that practically, 10,000 is the general cutoff that is used - it's small enough to be computationally feasible, big enough that it's a good simulation size. Over a broad range of applications I have found that your results don't change enough to matter whether you take ten thousand or ten billion runs of a simulation. So 10,000 it is.

The basic idea of all those runs is to see, to what extent does the variance in the prediction model (e.g. polling averages in this state or that state or national margin, etc) have an effect on the chance of a certain election result. The repeated running of the simulation generates enough runs of the possible variation in the results to get a good idea of what chances a candidate really has. But at the end of the day, the statement we're making is something akin to this: "The model was run with the parameters specified, and performed 10,000 simulations of the results. In 7501 of those runs, Hillary Clinton was victorious, in 2406, Donald Trump was victorious, and in 93 we had an electoral draw. So based on these results of my model, I am 75.01% sure that Hillary Clinton is going to win." Ultimately it is a probabilistic statement of belief, because it's a one-time event. You can't run an election more than once, it's a one-time event so the idea of "frequencies" is just not applicable. You are merely collecting evidence supporting a certain hypothesis and using simulations to model to what extent you are sure that that outcome will occur. And as new data is added, that evidence is updated - with new polls, new fundamentals data, and new pollster ratings. Bayesian through-and-through.

You will often find people talking about polls like "if we re-ran this election 100 times, we would expect Remain to win 52 times." The people who say that are actually just wrong. I certainly hope Nate Silver didn't say it exactly that way; I've generally been very impressed with how technically accurate he has been in his dialogues. If he did, either he was just making an analogy for saying "you can think of Trump's probability of victory as about this much" or he was just not being careful (for shame). But my description above is really how you should think about the probabilities in the context of the election predictions. And of course, those predictions change a lot based on how the evidence evolves. Trump's chances in the past week have increased substantially, and some recent articles from 538 have discussed that in some depth. There is not a "random chance" of Hillary Clinton and Donald Trump winning - we just don't know who will win. We have some idea, but we only have a general sense for how much the evidence points to one result or the other. At this moment in time, the 538 model is about 67% sure that the factors will line up in favor of a Hillary Clinton win. And that is the long answer for what these probabilities are actually saying.