A take on presenting predictions

Any prediction of the future carries some uncertainty. As if prediction wasn’t hard enough, estimating this uncertainty and presenting it in a way that makes sense is surprisingly hard too. No one gets more attention drawn to their mistaken predictions than weather forecasters, so it might be interesting to see how they’r tackling uncertainty. Recently yr.no, the primary supplier of weather forecasts here in Norway, did their own take on this problem. Each prediction in their long-term forecast is now colour-coded.  In the figure below, “green predictions” come true in more than 70% of the cases, yellow between 50% and 70% and red forecasts are correct in less than half of all cases.


In addition, there’s a nifty continuous plot of how the uncertainty in the forecasts on rainfall and temperature develop through the forecast period:


In this plot, reality will lay inside the dark grey and dark blue areas in 50% of cases and as far out as the light dark and light blue areas in 30% of cases. We can see how the temperature predictions get more and more uncertain further out in the forecast, a great way to visualize the {en:butterfly effect}! But the figure holds a small surprise too: The uncertainty in rainfall actually decreases further out in the prediction period! We also notice in the previous figure that accuracy in wind speed predictions picks up on the last day of the forecast.  This runs counter to our intuition, which is just why we need visualizations like these.

To be able to present the uncertainty of their models, Yr.no or the Norwegian Meteorological Institute which is behind it, had to rework their whole approach to weather forecasting. They went from running spaghettione model of the weather to running an ensemble of 51 slightly different simulations in parallell. The long term forecast is established by a majority vote between the simulations and the number of dissenting models gives the forecast uncertainty. Temperature is determined by the median value. The visualization is somewhat similar to a {en:Spaghetti Plots|spaghetti plot}, like in the picture to the right. (Click for larger version)

Yr also seems to take into account how well the models perform at a particular place. Some geographic locations are dominated by quick changes in the weather that are hard to predict, other places are handicapped by being far from weather stations. This is reflected as a systematic uncertainty in the long term forecast.

yr.no is hugely popular in Norway, which means that many here will be competent users of these kinds of plots.  This in turn means that other professions would have an easier time explaining the uncertainty in their forecasts if they used yr’s template. Is this feasible? The yr template has some shortcomings which may not matter much for weather predictions  but is crucial in other fields. Most obvious is perhaps how the unlikely outcomes are glossed over. There could be a 10% or 0.0001% chance of a tornado but you can’t tell from the plots. This cutoff is somewhat remeniscent of Taleb’s {en:Black swan theory} and his critique related to the financial crisis. But this is also a weakness of the ensemble approach. An event that has only 1% chance of occuring has only a 40% chance of appearing in any of the 51 simulations, so low-probability events cannot even be estimated reliably.

Another weakness is that you’ll never see a {en:Bifurcation theory|bifurcation} here. The plots of temperature and  precipitation assumes that the probability distribution has one maximum that tails off in both directions. The ensemble could, at least in theory have been split between say 25 simulations predicting a very hot day and 26  predicting a cold day, with no predictions in between. The plot however, would misrepresent this as a skewed distribution with “cold” as most likely and a wide tail up to “hot”.

Finally, the plots do not give you a sense of the joint probability distribution. Friday morning in the above plot could possibly be rainy or dry and it could be below or above freezing. But how likely is snow? The probability distributions are not independent so this can’t be answered by simply combining the two distributions in the plot.

We could of course modify the yr template slightly to address these issues, but would this confuse the user? Maybe anything beyond the red to green colour-coding is too much information. I’d love to hear of other ways to represent uncertainty that anyone has come across.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

2 Responses to A take on presenting predictions

  1. Pingback: Quora

  2. Pingback: Quora

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s