“Prediction,” the great physicist Niels Bohr is said to have once observed, “is very difficult. Especially when it concerns the future.” In science and economics, our lack of ability to foresee the future has traditionally been attributed to two theories – the butterfly effect, and the efficient market hypothesis – which have more in common than might appear.

The “butterfly effect” was first coined by the meteorologist Ed Lorenz in a 1972 talk, based on an earlier paper in which he observed that the solutions to a highly simplified weather model were sensitive to initial conditions. When he slightly changed the inputs to the model and ran the simulation, the answer changed completely. This was like running a weather prediction model using slightly different values for today’s weather, and getting wildly divergent forecasts for the weather next week.

Theorising repercussions
Lorenz’s model only consisted of a few equations, and was not intended to be realistic, but it showed in principle that systems such as the atmosphere, or for that matter the economy, might be very sensitive as well. As former Federal Reserve chairman Ben Bernanke put it, paraphrasing Lorenz, “a small cause – the flapping of a butterfly’s wings in Brazil – might conceivably have a disproportionately large effect – a typhoon in the Pacific.” Prediction of such an unstable and highly-strung system would clearly be impossible.

Economists have an advantage over weather forecasters, in that they also have another theory of non-prediction, based this time on the idea of efficient markets. This came out of a 1970 thesis by Eugene Fama, who defined an efficient market as one where “there are large numbers of rational profit maximisers actively competing, each trying to predict future market values of individual securities, and where important current information is almost freely available to all participants.” In such a market, he believed, “competition among the many intelligent participants” would drive prices towards their “intrinsic value” where all relevant information was taken into account.

Efficient markets are unpredictable because they are so sensitive that they instantly correct for any slight change in the economy

For example, if the price of a security exceeded its intrinsic value, investors would sell it, and the price would immediately revert to its correct level. And as new pieces of information arose – say earnings reports or an unemployment reading – the market would instantaneously take them into account. The markets knew everything that could be known, and the rest was random. It therefore followed that no one could beat the market – and no one could predict it. Techniques such as chart analysis (looking for recurrent patterns in market data) or fundamental analysis (for example looking for companies that are undervalued relative to earnings) were a waste of time according to Fama.

Empirical evidence would appear to agree. As economist John Cochrane wrote, “The surprising result is that, when examined scientifically, trading rules, technical systems and market newsletters have essentially no power beyond that of luck to forecast stock prices… The main prediction of efficient markets is exactly that price movements should be unpredictable!”

Efficiently unpredictable
So what do these theories have in common, apart from the fact that they both predict unpredictability, and emerged into the public consciousness in the early 1970s? The first thing is that they are both based on a magical degree of extreme sensitivity. The butterfly effect says that complex systems such as the atmosphere or the economy are unpredictable because they can be perturbed by minuscule changes. Efficient markets are unpredictable because they are so sensitive that they instantly correct for any slight change in the economy (including a change in the weather).

Both theories provide an explanation for our inability to predict the future, but still allow for probabilistic predictions. In the 1990s, weather forecasters began making multiple forecasts from slightly-perturbed initial conditions to get an idea of the range of possible future outcomes (a technique known as ensemble forecasting) and find out which are most likely. Similarly, the efficient market hypothesis implied that prices should vary randomly around an equilibrium value. This led to the development of tools such as the Black-Scholes formula for pricing options, or Value at Risk for assessing the probability of price changes.

Finally, both theories have a drawback, which is that they are wrong – or rather, they are right for the wrong reasons. While many equations are sensitive to initial condition, the butterfly effect was based on a toy mathematical model that had little relationship to the weather. As physicist Stephen Wolfram points out, viscous effects would tend to dampen out small perturbations in the real atmosphere, and experiments with weather models show them to be relatively insensitive to even large perturbations. Forecast error is due, not to butterflies or chaos, but to a simpler and less glamorous problem – model error. The atmosphere is a complex system that eludes precise simulation.

The efficient market hypothesis, meanwhile, is based on numerous assumptions such as the existence of rational investors who act independently from one another; but as behavioural economists have shown, real markets consist of people who are in constant interaction and are driven in large part by emotions, which is why they all tend to stampede to the exits at the same time during a market crisis. Rather than markets driving prices to their correct equilibrium levels, complexity scientists argue that it makes more sense to see markets as being at a state that is far from equilibrium, and susceptible to sudden changes. Crashes are hard to predict for the same reason that earthquakes are hard to predict.

So why are these theories still around? The traditional test of a scientific theory is that it should be able to make accurate predictions based on a logical, mechanistic argument. While the butterfly effect and the EMH correctly “predict” that systems are unpredictable, however, one needs to be careful about predictions that only give an explanation for something that is already known. As Bohr said, predictions about the future are much harder.

A simpler, if less attractive, explanation for our inability to predict our future is that systems such as the economy, or the atmosphere, are too complex to be captured by a mathematical model. Of course, this would mean that economists would have to ditch techniques such as Value at Risk – but given that those methods all failed during the recent crash anyway, maybe that is no bad thing.