Oscar prognosticators: how did they do at picking the Academy Award winners?

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div id="article-header" style="background-color: white; background-repeat: no-repeat no-repeat; border-collapse: collapse; border-color: rgb(209, 0, 139); clear: left; color: #333333; font-family: arial, sans-serif; font-size: 12px; line-height: 15px; margin: 0px; min-height: 68px; padding: 0px; position: relative;"><div id="main-article-info" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; float: left; margin: 0px; padding: 0px;"><h1 itemprop="name headline " style="background-repeat: no-repeat no-repeat; border-bottom-color: rgb(209, 0, 139); border-collapse: collapse; border-left-color: rgb(209, 0, 139); border-right-color: rgb(209, 0, 139); border-top-width: 0px; font-family: georgia, serif; font-size: 2.166em; font-weight: normal; line-height: 1.154; margin: 0px 0px 2px; padding: 0px; width: 460px;"><span style="color: #666666; font-family: arial, sans-serif; font-size: 1.333em; line-height: 1.25;">Nate Silver and others took their statistical analysis to the film industry, but predicting awards might be tougher than elections</span></h1></div><ul class="share-links trackable-component" data-component="Article:top share tools" id="content-actions" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; float: right; list-style: none; margin: 5px 0px 0px; overflow: hidden; padding: 0px; width: 140px;"><li class="full-line facebook" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 5px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding: 0px;"><span class="facebook-share" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: block; font-size: 0.88em; margin: 0px; overflow: hidden; padding: 0px;"><a class="facebook-share-btn" data-href="http://www.guardian.co.uk/film/filmblog/2013/feb/25/oscar-prognosticators-academy-award-winners" data-link-name="Facebook Share" href="http://www.facebook.com/dialog/feed?app_id=180444840287&link=http://www.guardian.co.uk/film/filmblog/2013/feb/25/oscar-prognosticators-academy-award-winners&display=popup&redirect_uri=http://static-serve.appspot.com/static/facebook-share/callback.html&show_error=false" style="background-color: #edeef4; background-repeat: no-repeat no-repeat; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-collapse: collapse; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(205, 213, 229); color: #3b4d7b; cursor: pointer; float: left; height: 14px; margin: 0px; padding: 2px 4px; position: relative; text-decoration: none;"><span class="facebook-share-icon" style="background-image: url(http://static.guim.co.uk/static/fbbd5444dca1eb9b96ba971c352038f24cff1be2/common/styles/images/btn_facebook_12x12.png); background-position: 0px 1px; background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline; float: left; height: 13px; margin: 0px 2px 1px 1px; padding: 0px; width: 13px;"></span><span class="facebook-share-label" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #3b5998; line-height: 1.3; margin: 0px; padding: 0px;">Share</span></a><span class="facebook-share-count" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border: 1px solid rgb(205, 213, 229); float: left; height: 14px; margin: 1px 0px 0px 6px; padding: 1px 2px; position: relative;"><i style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border-color: transparent white transparent transparent; border-style: solid; border-width: 4px 5px 4px 0px; display: block; height: 1px; left: -3px; margin: 0px; padding: 0px; position: absolute; top: 4px; z-index: 5;"></i><u style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border-color: transparent rgb(215, 215, 215) transparent transparent; border-style: solid; border-width: 4px 5px 4px 0px; display: block; height: 1px; left: -5px; margin: 0px; padding: 0px; position: absolute; top: 4px;"></u>0</span></span></li><li class="full-line" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 5px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding: 0px;"><iframe allowtransparency="true" class="twitter-share-button twitter-count-horizontal" data-twttr-rendered="true" frameborder="0" scrolling="no" src="http://platform.twitter.com/widgets/tweet_button.1360972506.html#_=1361896004851&count=horizontal&counturl=http%3A%2F%2Fwww.guardian.co.uk%2Ffilm%2Ffilmblog%2F2013%2Ffeb%2F25%2Foscar-prognosticators-academy-award-winners&id=twitter-widget-0&lang=en&original_referer=http%3A%2F%2Fwww.guardian.co.uk%2Ffilm%2Ffilmblog%2F2013%2Ffeb%2F25%2Foscar-prognosticators-academy-award-winners&related=GuardianFilm&size=m&text=Oscar%20prognosticators%3A%20how%20did%20they%20do%20at%20picking%20the%20Academy%20Award%20winners%3F&url=http%3A%2F%2Fgu.com%2Fp%2F3e3td%2Ftw&via=guardian" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; height: 20px; margin: 0px; padding: 0px; width: 116px;" title="Twitter Tweet Button"></iframe></li><li class="full-line google-plus" data-link-name="Google plus" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 2px !important; margin-left: 0px; margin-right: 0px; margin-top: -2px !important; padding: 0px;"><div id="___plusone_0" style="background-color: transparent; border-collapse: collapse; border-style: none; display: inline-block; float: none; font-size: 1px; height: 20px; line-height: normal; margin: 0px; padding: 0px; vertical-align: baseline; width: 90px;"><iframe allowtransparency="true" data-gapiattached="true" frameborder="0" hspace="0" id="I0_1361896002852" marginheight="0" marginwidth="0" name="I0_1361896002852" scrolling="no" src="https://plusone.google.com/_/+1/fastbutton?bsv&size=medium&hl=en-GB&origin=http%3A%2F%2Fwww.guardian.co.uk&url=http%3A%2F%2Fwww.guardian.co.uk%2Ffilm%2Ffilmblog%2F2013%2Ffeb%2F25%2Foscar-prognosticators-academy-award-winners&jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.zh_TW.4dkcFihTIko.O%2Fm%3D__features__%2Fam%3DgQ%2Frt%3Dj%2Fd%3D1%2Frs%3DAItRSTPjvWFCmxksF4tkwTQBQLNe2q_nJw#_methods=onPlusOne%2C_ready%2C_close%2C_open%2C_resizeMe%2C_renderstart%2Concircled%2Conload&id=I0_1361896002852&parent=http%3A%2F%2Fwww.guardian.co.uk&rpctoken=37237204" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border-style: none; height: 20px; left: 0px; margin: 0px; padding: 0px; position: static; top: 0px; visibility: visible; width: 90px;" tabindex="0" title="+1" vspace="0" width="100%"></iframe></div></li><li class="full-line linked-in" data-link-name="LinkedIn" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 5px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding: 0px;"><span class="IN-widget" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline-block; line-height: 1; margin: 0px; padding: 0px; text-align: center; vertical-align: baseline;"><span style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline-block !important; font-size: 1px !important; margin: 0px !important; padding: 0px !important; vertical-align: baseline !important;"><span id="li_ui_li_gen_1361896003446_0" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: block !important; margin: 0px; overflow: visible !important; padding: 0px; position: relative !important;"><a href="http://www.blogger.com/blogger.g?blogID=8896966065785019252" id="li_ui_li_gen_1361896003446_0-link" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border: 0px !important; color: #005689; display: inline-block !important; height: 20px !important; margin: 0px !important; padding: 0px !important; text-decoration: none !important;"><span id="li_ui_li_gen_1361896003446_0-logo" style="background-image: url(http://static.licdn.com/scds/common/u/img/sprite/sprite_connect_v13.png) !important; background-position: 0px -276px !important; background-repeat: no-repeat no-repeat !important; border-collapse: collapse; border: 0px !important; cursor: pointer !important; display: block !important; float: right !important; height: 20px !important; left: 0px !important; margin: 0px !important; overflow: hidden !important; padding: 0px !important; position: absolute !important; text-indent: -9999em !important; top: 0px !important; width: 20px !important;">in</span><span id="li_ui_li_gen_1361896003446_0-title" style="background-color: rgb(236, 236, 236) !important; background-image: -webkit-linear-gradient(top, rgb(254, 254, 254) 0%, rgb(236, 236, 236) 100%) !important; background-repeat: no-repeat no-repeat; border-bottom-color: rgb(185, 185, 185) !important; border-bottom-left-radius: 0px !important; border-bottom-right-radius: 2px !important; border-bottom-style: solid !important; border-collapse: collapse; border-right-color: rgb(191, 191, 191) !important; border-right-style: solid !important; border-top-color: rgb(226, 226, 226) !important; border-top-left-radius: 0px !important; border-top-right-radius: 2px !important; border-top-style: solid !important; border-width: 1px 1px 1px 0px !important; color: rgb(51, 51, 51) !important; cursor: pointer !important; display: block !important; float: left !important; height: 18px !important; line-height: 20px !important; margin-bottom: 0px; margin-left: 1px !important; margin-right: 0px; margin-top: 0px; overflow: hidden !important; padding: 0px 4px 0px 23px !important; text-shadow: rgb(255, 255, 255) -1px 1px 0px !important; vertical-align: top !important; white-space: nowrap !important;"><span id="li_ui_li_gen_1361896003446_0-mark" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline-block !important; margin: 0px; overflow: hidden !important; padding: 0px; width: 0px !important;"></span><span id="li_ui_li_gen_1361896003446_0-title-text" style="background-color: transparent !important; background-image: none !important; border-collapse: collapse; display: inline-block !important; float: none !important; font-family: Arial, sans-serif !important; font-size: 11px !important; font-weight: bold !important; height: 18px !important; margin: 0px; padding: 0px; vertical-align: baseline !important;">Share</span></span></a></span></span><span style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline-block !important; font-size: 1px !important; margin: 0px !important; padding: 0px !important; vertical-align: baseline !important;"><span class="IN-right" id="li_ui_li_gen_1361896003473_1-container" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; cursor: pointer !important; display: inline-block !important; float: left !important; height: 18px !important; margin: 0px; overflow: visible !important; padding-bottom: 0px; padding-left: 2px !important; padding-right: 0px; padding-top: 0px; position: relative !important;"><span class="IN-right" id="li_ui_li_gen_1361896003473_1" style="background-image: url(http://static.licdn.com/scds/common/u/img/sprite/sprite_connect_v13.png) !important; background-position-x: 100%; background-position-y: -100px !important; background-repeat: no-repeat no-repeat !important; border-collapse: collapse; display: block !important; float: left !important; height: 18px !important; margin-bottom: 0px; margin-left: 0px; margin-right: 4px !important; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 4px !important; padding-top: 0px;"><span class="IN-right" id="li_ui_li_gen_1361896003473_1-inner" style="background-image: url(http://static.licdn.com/scds/common/u/img/sprite/sprite_connect_v13.png) !important; background-position: 0px -60px !important; background-repeat: no-repeat no-repeat !important; border-collapse: collapse; display: block !important; float: left !important; margin: 0px; padding-bottom: 0px; padding-left: 8px !important; padding-right: 0px; padding-top: 0px;"><span class="IN-right" id="li_ui_li_gen_1361896003473_1-content" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; display: inline !important; font-family: Arial, sans-serif !important; font-size: 11px !important; line-height: 18px !important; margin: 0px; padding: 0px 5px !important;">0</span></span></span></span></span></span></li><li class="full-line email" data-link-name="email this story" style="background-image: url(http://static.guim.co.uk/static/fbbd5444dca1eb9b96ba971c352038f24cff1be2/common/styles/images/bg-email.png); background-repeat: repeat no-repeat; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-collapse: collapse; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); height: 18px; margin-bottom: 5px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding: 0px; width: 60px;"><a class="rollover send-email" href="http://www.guardian.co.uk/film/filmblog/2013/feb/25/oscar-prognosticators-academy-award-winners#" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #333333; float: left; line-height: 1.5; margin: 0px; padding: 0px; text-decoration: none; width: 60px;" title="Send to a friend"><img alt="" class="trail-icon" src="http://static.guim.co.uk/static/fbbd5444dca1eb9b96ba971c352038f24cff1be2/common/images/icon-email.png" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border: none; margin: 0px 3px -2px 4px; padding: 0px;" />Email</a></li></ul></div><div id="content" style="background-color: white; background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #333333; float: left; font-family: arial, sans-serif; font-size: 1.166em; line-height: 1.357; margin: 0px 0px 10px; padding: 0px; width: 460px; z-index: 2;"><div data-component="Article:articlebody" id="article-wrapper" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; border-color: rgb(209, 0, 139); border-top-style: solid; border-top-width: 1px; margin: 0px; padding: 5px 0px 0px; position: static !important;"><div id="main-content-picture" itemprop="image" itemscope="" itemtype="http://schema.org/ImageObject" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin: 0px 0px 14px; padding: 0px;"><img alt="christoph waltz oscar" height="276" itemprop="contentUrl representativeOfPage" src="http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2013/2/25/1361833448877/christoph-waltz-oscar-008.jpg" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin: 0px; padding: 0px;" width="460" /><div class="caption" itemprop="caption" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #666666; font-size: 12px; margin: 0px; padding: 0px;">All but Farsite had Tommy Lee Jones winning best supporting actor – Farsite correctly picked Christophe Waltz. Photograph: Christopher Polk/Getty Images</div></div><div id="article-body-blocks" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin: 0px; padding: 0px;"><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">While Oscar fans spent Monday checking their pools to see if they picked more winners than their friends, I was wondering whether statistics experts did a good job forecasting the results.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">I found four websites that used a <a href="http://www.popsci.com/science/article/2013-02/best-mathematical-models-for-predicting-the-oscars" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">variety of statistical methods</a> to predict winners. <a href="http://oscarforecast.wordpress.com/predictions/" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">Ben Zauzmer</a>, a Harvard sophomore, built a model from critics' ratings and results from other awards shows. <a href="http://www.farsiteforecast.com/vanity-fair-polls-and-parties/" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">Farsite</a> <a href="http://www.reuters.com/article/2013/01/10/oh-farsite-oscars-idUSnPnCL40497+160+PRN20130110" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">used data such as</a>results from other awards shows, the total number of nominations, buzz, and prior nominations for actors and actresses. <a href="http://fivethirtyeight.blogs.nytimes.com/2013/02/22/oscar-predictions-election-style/" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">Nate Silver</a> used awards shows as a kind of <a href="http://dartthrowingchimp.wordpress.com/2013/02/25/a-quick-post-mortem-on-oscars-forecasting/" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">pre-election poll average</a>. <a href="http://www.predictwise.com/node/1298" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">PredictWise</a> combined prediction markets and user sentiment from games.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">All four are at least somewhat affected by past awards shows. The first three make an index from previous awards, while PredictWise puts a lot more emphasis on what everyday people (whether through betting markets or games) thought – and obviously they must be heavily influenced by previous awards shows, too.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">None of them made any big misses. They all agreed on <a href="http://www.guardian.co.uk/film/danieldaylewis" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;" title="More from guardian.co.uk on Daniel Day-Lewis">Daniel Day-Lewis</a>for best actor, Jennifer Lawrence for best actress, Argo for best film and Anne Hathaway for best supporting actress.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">The categories where they disagreed is where the errors occurred. All but Farsite had Tommy Lee Jones winning best supporting actor for Lincoln. Farsite correctly pegged Christophe Waltz in Django Unchained.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">Those who missed here recognized the difficulty. PredictWise had Jones only at 44% to win with Waltz just behind at 40%. Zauzmer put Jones at 43% and Waltz at 34%. Silver's miss was worse than either PredictWise in that he had Philip Seymour Hoffman in The Master just slightly ahead of Waltz in probability. The reason seems to have something to do with Hoffman garnering more nominations for his role in other contests. Even so, Silver still called it the "most competitive category".</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">The worst miss in the major six was best director. Credit must go to Zauzmer who had <a href="http://www.guardian.co.uk/film/anglee" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;" title="More from guardian.co.uk on Ang Lee">Ang Lee</a> for Life of Pi as the favorite. The rest had<a href="http://www.guardian.co.uk/film/stevenspielberg" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;" title="More from guardian.co.uk on Steven Spielberg">Steven Spielberg</a> for Lincoln. I say "worst" miss because both Farsite and PredictWise had Spielberg at over 75% to win. The miss was particularly bad for Farsite because it tended to be quite conservative in its percentage chances to win and only had Hathaway at a greater chance to win an award. Silver, to his credit, called it a tossup for lack of data.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">Thus, the final scores were Farsite and Zauzmer five for six, and PredictWise and Silver four for six. I really wouldn't read very much into these small differences. It's a very small sample size, not significant to make a judgement about which models are better predictors or better calibrated (ie understanding uncertainty).</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">When we expand to the 21 categories prognosticated by both Zauzmer and PredictWise, they score evenly at 17 out of 21. Both had Lincoln third in best production design, even though it won. In addition, Zauzmer did not get Django Unchained in best original screenplay or Skyfall for best cinematography, while PredictWise failed to peg Les Misérables in best makeup and hairstyling.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">A more important point to take away is that there was at least one statistical predictor got it right in all six major categories. That suggests that a key fact about political forecasting holds for the Oscars: averaging of the averages works. You get a better idea looking at multiple models, even if they themselves include multiple factors, than just looking at one.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">This <a href="http://marginoferror.org/2012/11/08/aggregating-the-aggregates/" style="background-repeat: no-repeat no-repeat; border-collapse: collapse; color: #005689; margin: 0px; padding: 0px; text-decoration: none;">average of the averages</a> held among the wider 21 categories. In only one, best production design, did one of the four not get it. A little over 95% of the time did we have at least some idea of what was coming. This is pretty comparable to election polling.</div><div style="background-repeat: no-repeat no-repeat; border-collapse: collapse; margin-bottom: 13px; padding: 0px;">The overall lesson here is that while we won't always know the winner from any one model, we will rarely be too surprised so long as we take into all the statistical models. My guess is that 17 out of 21 or about 80% is a pretty good guide of what the best systems can do. There's no sign that anyone has cracked any sort of code here. They all did about as well and together gave us a very good heads-up as to who would win.</div></div></div></div></div>

2013 Oscar Oscar prognosticators: how did they do at picking the Academy Award winners?

Nate Silver and others took their statistical analysis to the film industry, but predicting awards might be tougher than elections Share 0 i...

Read more »