Effectively all of our understanding of finance and investing is based, at least to some degree, on historical returns. But how good of a guide is the past? Is it actually different this time?
We’ve already talked about the limitations of models in how we understand investing, and how we use statistics to analyze what’s going on in the markets.
But at an even more fundamental level, how useful is all of the historical returns data that we are swamped with? Does knowing what happened in the past give us any useful information about what might happen in the future – and more to the point, can you use that information to help you create your retirement income plan?
To spoil the story, yes the historical data is incredibly useful. But you need to be very cautious in how you use it, and be careful about the conclusions that you draw from it.
The Past is a Foreign Country
To understand why looking at the past is useful for understanding what might happen in the future, it’s helpful to start with the problems with looking at historical data.
Simply put, the world is different than it used to be.
Currently, we have roughly 100 years of reliable historical market data in the US. People were still people (with everything that statement implies) back in 1920s, but the way the world operated was wildly different. The financial, regulatory, and tax landscapes were all wildly different than they are today. Back then, the railroads were the high tech stocks of the day (though they were just coming out of a pandemic as well, so maybe I don’t want to push the differences too far).
And even if we expand this a little bit, the 20th century was a phenomenally good century for the US. Throughout the 20th century the US was one of, if not the, dominant country and economy in the world. And the US has had a pretty clear upward trajectory since the start of our historical market returns. In fact, even during the period when the US was actively involved in World War 2 (December 1941 – September 1945), the S&P 500 Index had an annualized return of nearly 23% (to put this in perspective, from 1926-2023 the annualized return was “only” 10%). While pretty much all of the other major countries were systematically destroying each other throughout the world. We don’t have good returns data on the German or Japanese (or UK, or France, or Italian, or…) stock markets, but investors in the rest of the world definitely weren’t seeing returns over 20% per year.
How likely is it that we will see another century like the 20th in terms of economic stability and growth? I’d imagine it’s pretty unlikely. And if we have materially different conditions – which play a large role in overall market returns – how likely are we to see the same sort of results that we have seen in the past?
So Why Look at Historical Data?
If the world is significantly different from the world where we saw a lot of these historical returns, why look at them?
To a certain extent, because that’s what we have.
There’s really no other way to estimate what future returns will look like. Even the fancy models that they gush over on CNBC (and which are usually wrong) are, at root, based on historical data.
But how we use the historical data is incredibly important. While many of us may wish for a precise answer, it’s important to recognize that this simply does not exist for pretty much anything meaningful in the financial markets. There’s simply too much noise.
Are Long Term or Short Term Expectations Better? No.
A good way to think about using historical data is that you’re thinking about a distribution of potential results. For instance, this is the distribution of the annual returns of the S&P 500 Index from 1926 through 2023.
We had one year with a return below -40% (1931), and two years with returns greater than 50% (1933 and 1954), and a whole mess of returns everywhere in between. Most years cluster roughly where you would expect them to – slightly more than 50% of years had returns between -5% and 25%. But even that is a pretty wide range.
But we can simplify this a little bit. If we keep in mind how big the distribution of returns is, we can think about what the most likely return is over the next year. So, again using the S&P 500 Index, if you were to ask what return I “expect” in 2024, I would say about 12% – because that is the long term average annual return for the S&P 500 Index.
But I would be absolutely shocked if we exactly hit the average return (12.16% to be precise). The dispersion of the returns is really big (and this is also why you shouldn’t expect your investment returns to converge through time).
How Stable is the Data?
But I don’t want to undersell the value of the data here. No one is going to accuse the stock market of being a model of stability (because it’s really not), but it’s long term average returns are remarkably consistent.
As we said earlier, the annual average return of the S&P 500 Index from 1926-2023 was 12.2% per year. But what if we cut that period in half. Do the two periods have similar average returns?
In essence, if we were sitting at the midpoint of the data, could we use what we saw in the first half to get a good picture of what might happen in the second half?
Well, yeah.
The annual average return of the first half of the data (from 1926-1974) was 10.9% per year. Over the second half (from 1975-2024) the annual average return was 13.5% per year. That difference of 2.6% per year is a reasonably big number but remember that the standard deviation of the S&P 500 over the full period was about 20% per year. And with that sort of variance, it is very likely that this is just random noise. So in this case, it’s pretty fair to say that the historical data gave you a pretty good estimate of what would happen next.
But understanding what the historical outcomes have looked like can give us a pretty decent set of what the future outcomes might look like.
Understanding the Relationships in the Data
While a large part of the way that we use historical data is to estimate capital market assumptions (the fancy way of saying what we think different asset classes will do going forward), another big way that we use the historical data is to understand the relationships at work in the markets. These relationships are critical as we think about how to structure our portfolio, and specifically think about the risk and return tradeoffs for the different risk factors as we move away from the market portfolio looking for either higher returns, or lower volatility.
While the returns of these risk factors can, and do, change through time, the historical data is crucial for understanding how they work. For instance, the expected outperformance of stocks over bonds may change through time, but it’s hard to argue that stocks aren’t going to be riskier and have a higher expected return than bonds. Which is exactly what we see in the data.
By starting with the data, and then looking at the underlying logic (as well as making sure that the data is consistent through time and across different markets), we can decrypt a lot of what the market is telling us over the long term. And this process is crucial for deciding what risks we do, and do not, want to take on.
Should You Use Historical Data?
So, should you use historical data in your planning?
Yes.
But you need to do so with the understanding that there is no precision in the data. Financial returns are noisy, and strange things happen. You need to take the historical data with a grain of salt (and a pretty big helping of humility), but understanding the past gives you access to a treasure trove of data about what the future might look like – and how that might impact your retirement income plan.
To find out more about how to build an investment portfolio that works for you, read our eBook 9 Principles of Intelligent Investors.
