In my experience the fundamental basis behind nearly all winning handicapping strategies is a correct understanding of how regression to the mean applies to the particular sport and league in question. The concept is so important that one could write a several hundred page sports handicapping book on the concept alone. Perhaps we will do that one day. But for today we will discuss what regression to the mean is, what causes it in sports analytics, and how one can begin to apply the concept to their handicapping, even for those who do not rely on statistical models.
There are two types of regression to the mean in sports analytics. The first is what I think most people associate with the term, statistical regression to the mean. Statistical regression to the mean has its basis on the fact that sporting outcomes are subject to a great deal of random chance. Even with identical players and conditions, a sporting event over some time period - whether it is a single shot, a game, or a season - is not going to have the same outcome every time. In modern sports luck always plays a role. Particularly when facing similar-level competition, humans never have the required level of skill to make every shot, throw every pitch perfectly, catch every ball, etc. And even when they do have this required level of skill, game theory is also important. Aspects of certain sports such as pitch selection in baseball and play calling in football/basketball always ensure that luck will be present.
The result of this element of random chance in sports is that when measuring any sports statistic, one will always observe extreme outcomes, as with enough players and teams, some will always be luckier than others. The concept of regression to the mean states that any extreme performance is likely at least due in part to this random chance. Knowing this is the case, the future performance can always be expected to be closer to the mean value of the entire statistic in the future than it was in the past.
A second type of regression to the mean in sports is what I call structural regression to the mean. While there is always random variation that leads to extremes in performance, in nearly every sport, there are structural elements at work that cause extreme performers to regress towards the mean even in the absence of any random chance.
Let's take an example of a top-performing pitcher in baseball, who had a 2.00 ERA in the past season (ERA is not a great statistic to evaluate pitchers, but is fine for the purposes of this article). Let's further assume that this pitcher was not lucky - he actually was a 2.00 ERA pitcher last season, that is, if he pitched 100 seasons with the same level of ability he had last year, he would average 2.00 in those 100 seasons.
The question is, how do we think this pitcher will perform this season? The answer is almost always going to be "higher than 2.00". And the reason is simply "there are more ways for a pitcher at this level to get worse than better". For a top-performing pitcher, it is very difficult to improve. They are not going to add much velocity or command as these had to be top-class to be a 2.00 ERA pitcher in the first place, plus players tend to get worse over time due to aging. Even if they develop a great new pitch, it is hard for it to be better than the pitches they already have. They could improve their pitch sequencing, but it was probably already really good.
On the other hand, there are many ways for this pitcher to get worse. They are definitely going to get older, so they will probably lose velocity. They could get injured. They could sign a new contract based on their performance and lose motivation. But even if they are just as good, they may have also been a 2.00 ERA pitcher in the previous season because they relied on pitches or strategies that hitters were not familiar with. With another season to prepare, these hitters will inevitably begin to catch on to what the pitcher is doing.
The same balance applies to the other end. If a pitcher is really bad, say a 5.00 ERA, they will almost always improve the following season, if they continue to pitch. The reason is that if they don't get better, they probably won't stay in the league. And if they do stay in the league, even if they don't perform better right away, there is probably some reasonable expectation, from a coach or someone else making the decisions, that they have improved.
Now this is slightly different than the case of the elite pitcher, because there is no intrinsic reason for this pitcher to actually get better, whereas with the elite pitcher, there are intrinsic reasons why they should get worse. However, it is easier for bad players to improve than good players. They will tend to be more motivated to perform as the financial incentives are greater. They may have the opportunity to copy the strategies or learn the game from better players (especially for rookies). So more bad players are able to improve, and those that do not improve fall out of the population, leading to a regression to the mean.
Structural regression to the mean does not just take place on the individual player level. There are structural factors in team performance that tend to pull teams together over time as well. Beyond obvious factors like the salary cap and free agency, teams that do badly will tend to change their way of doing business, often copying those teams which were successful in recent times. This leads to an improvement in the bad teams. On the other side, the more successful teams will often get complacent, which can impact their performance. Furthermore, they face being copied by other teams, which means they will have to constantly evolve to stay on top, which may or may not be difficult depending on how they got there.
The important question is, as handicappers, how do we account for regression to the mean in our handicapping? The key concept to realize is that some aspects of team performance are highly repeatable over short samples, while others require a full season or more to make any conclusions. The best way to figure this aspect out - how valuable these statistics are and how much they should be regressed - is statistical modeling, but you can get started just reading sports analytics articles on the internet. And in many cases, we don't need or don't have a performance sample to work with, and are better off looked at qualitative characteristics to decide which team is likely better. We can then build on that with statistics over time.
As an example, consider the case of two teams in a new football league, who have never played any other games in any other league. One team won their first game 42-14, and the other won 23-21. It is now our job to handicap these two teams as they face one another in week 2. The team who won 42-14 in their last game is at home.
Taken at face value, we could make the home team a 29 point favorite, as they won their last game by 28 vs the other team's 2, plus they are getting a 3 point home advantage, and we know nothing else about the teams. Of course, this would obviously be terrible. These teams are both in a brand new league but were probably created in a very similar manner; there is no reason to believe they are really that diffrerent, and 26 points before home advantage is a huge margin in a football game. So we'd want to regress the teams performance to the mean somewhat. One option, which wouldn't be the greatest, would be to assume the league was simliar to the NFL. We could then look at how teams performed in week 2 based solely on week 1 scoring margin. Just taking a guess, from the NFL we'd probably assume the home team is something like a 5 point favorite based just on these week 1 results. We'd then use our football knowledge to realize that the NFL is a league where many structural factors make teams similar over time, and our new football league is probably not like that. So maybe we'd make the home team more like a 7 or 8 point favorite instead.
But of course, there is more information in a football game than just score, and more information about a football team than just results of their games. To really find out if one team is better than the other from a single game, we could look at higher-frequency stats, such as yards per play. It is well known in football analytics that yardage is a more repeatable statistic over the long term than points or turnovers; in other words, it does not have to be as sharply regressed to the mean to predict future performance. In our example, if the home team heavily outgained their opponent while the other team barely won due to some lucky big plays and turnovers, we'd probably make the home team something more than a 7 point favorite in this game.
And if we wanted to be really sharp, sharp enough to actually win, we'd probably go into detail about the individual roster makeup of the two teams and their coaching staff, and probably also do the same for their week 1 opponents. One game, while all we have to go on, is never enough to statistically handicap anything. So we'd want to form a prior belief on how good each team is likely to be, based on factors like size and speed of the players, throwing ability of the quarterback, experience of the coaching staff, how much money each team had to spend on players, and the like. From there we'd probably get some idea which team is likely to be better. The challenge then might be to figure out how much each of these variables translates into a spread or total. We might be able to pull in data from other football leagues where we have experience to get an estimate, or we might just have to make a total guess; either way, we'd probably do better than if we just assumed the two teams were totally the same.
We'd then combine both factors - the most predictive statistics we have from the first game they played as well the expected strength of both teams and their week one opponents from our qualtiative evaluation - to make our final prediction. We'd still have to bring each team's performance level down to some mean, but the "mean level" would be better informed than simply assuming all teams are equal to start out. As the season went on, we'd begin to throw away our prior - that we got simply by looking at personnel - and begin to rely more on the actual results of games and their underlying statistics.
Properly weighting all the possible factors to come up with a single prediction is the great challenge of sports handicapping, and forecasting in general. Millions await those who can master this concept.