How Difficult Should the U.S. Open Be?

Every June, like clockwork, golf pros complain that the just-played U.S. Open course was too difficult. All sorts of unkind statements are made about how the USGA, as the pros say, takes a perfectly good course and makes it impossible to play. They might say “it was tricked up.”

Former USGA President Frank “Sandy” Tatum said in 1974, “We are not trying to embarrass the world’s best golfers; we are trying to identify them.” The simple fact is that the U.S. Open is different – and should be different – than other golf competitions. After all, it is a championship, not a tournament. And it is a national championship.

The reasoning behind the USGA’s perhaps overly difficult course setup is that it is assumed that the more difficult the course, the more likely the event will identify the best golfer – the golfer who can excel at all facets of the game. Is that assumption correct? Can that assumption be tested statistically or otherwise? This paper is an initial, far from perfect, attempt to examine if difficulty equates to better identification.

Golf fans seem to enjoy the very difficult conditions, perhaps because they like to see the pros struggle the way they themselves do when playing at their local course. It is safe to say that most pros appreciate what the USGA is trying to do and would be disappointed if the course were too easy and did not identify a “champion” golfer as the winner. Certainly, the winner never complains. To a certain extent, the severe conditions are what makes it the U.S. Open, and anything less would be too much like a regular tour event.

How “Difficult” is the U.S. Open?

The USGA has said that it would like to “protect par.” For this analysis, “difficulty” will be measured by the winning score as compared to par. The lower the winning score, the “easier” the course, and vice versa. Admittedly, this is not a perfect measure of difficulty. Clearly, there are examples when this has been true. In 1974, the winning score was 7 over par, and the event was labeled “Massacre at Winged Foot.” In 2017, the winning score at Erin Hills was -16, and many thought the course was too easy. On the other hand, no one would say Pebble Beach was too easy when Tiger Woods won there in 2000 with a score against par of -12, nor in 1980 when Jack Nicklaus won with -8.

The average score at the U.S. Open is regularly 5 to 10 strokes higher (read: harder) than regular PGA Tour events, even for the premier tour events on the best venues. The typical average score on the tour is from the high 60s to the low 70s. The average at the U.S. Open ranges from 74 to 77, and scores skew higher on regularly used PGA Tour venues (when used for the U.S. Open). Anecdotally, Riviera, Congressional and Olympic PGA Tour winning scores are generally in the -8 to -20 range. As noted, the USGA tries to set up the course to “protect” par.

Even among majors, the U.S. Open is more “difficult” as measured by the winner’s final score against par. Both the Open Championship (since 1959) and American PGA Championship (since 1958) have an average winner’s score of -8, whereas the U.S. Open (since 1946) has had an average winner’s score of -2. Are the higher winners’ scores at the U.S. Open due to the difficulty of the course, the pressure of the National Championship or a combination of both?

The U.S. Open setup is truly unique. The Masters Tournament is always played on the same course. Its committee knows pretty much how the course plays and sets it up to provide a wonderful challenge to the players as well as great entertainment to the fans. The R&A returns regularly to the same courses and has a better opportunity to predict how a course will play – their biggest obstacle is the weather. Often, their setups are easier to compensate for the possibility of severe weather conditions making the course insurmountable. The PGA Championship is believed to be somewhat more benign than the U.S. Open, although it is often played the same venues in hopes that their championship is distinguishable from regular tour events.

There is a concern that perhaps at times, the setup is too severe. How severe should it be? Does making it so severe increase or decrease the chances of identifying the best golfer? How different is the U.S. Open from the other majors in identifying the best golfer? What is the optimal setup for the U.S. Open?

There is no easy way to decide this issue, but modern analytics and statistics may help. The following is an attempt to bring a statistical/analytic approach to answering the question: Do harder setups increase the identification of the best player? This first effort is an “attempt” and limited in many ways. The assumptions behind it are open to challenge. Alternative approaches are perhaps better. But it is a first attempt in the hopes that much is revealed, and other analysts are encouraged to advance the science and move beyond this first try.

What Makes the U.S. Open More Difficult?

Golf is unique among sports in that its venue changes and can be changed. For the World Series, Major League Baseball doesn’t move the fences back. For the Super Bowl, the NFL doesn’t move in the sidelines. For the NBA Championship, they don’t raise the basketball hoop. For the U.S. Open, the USGA indeed does the equivalent of all of those.

For the tennis U.S. Open, the USTA doesn’t make their Open surface slicker than the surfaces at other events. Because the tennis court is the same, tennis majors are made more difficult by playing five-set matches instead of the usual three. (Should the golf U.S. Open be a five-round event?) Although they play more sets in tennis, it is likely that the big difference is the challenge of having to play your best when it matters the most, which sets their championship apart from regular tennis tour events. But unlike tennis, each golf course is different and can be set up in different ways. In golf, the course can be made more difficult.

To begin, to separate the best from the merely terrific golfers, the USGA picks courses that are generally considered the best courses in the country (read: very difficult). Then, the USGA purposely makes those venues uniquely difficult to make the U.S. Open stand as the ultimate test of golf at which only the very best will meet the challenge. The USGA wants to set the U.S. Open apart from the other majors and very much apart from the PGA Tour’s regular venues.

Traditionally, the fairways are narrower at the U.S. Open, often as narrow as 22 yards wide. In recent years, the USGA has substantially widened the fairways to keep the driver in the hands of most players. But the players still have to contend with sloping fairways, which tend to reject all but the absolutely most accurate tee shots, and sometimes even some of those.

The rough typically is much higher and thicker. At the 1980 U.S. Open at Baltusrol, I calculated the “penalty” for missing the fairway. The difference in average score on the par-4 holes for players whose tee shot was in the fairway versus for those in the rough was half a stroke, less on shorter par-4 holes and more on the longer ones. On the PGA Tour, the average difference is a quarter stroke. In recent years, the USGA has created two levels of rough in an effort to “penalize” less for a tee ball just off the fairway. Still, the conditions are more severe than on the tour.

The bunkering on the courses picked (e.g., Oakmont) generally have more, deeper, and more strategically placed sand hazards. The USGA generally does little with these. They are often too tough to begin with. But there have been times when bunkers have been added or made more difficult. (e.g., Tatum’s bunker on 16 at Pebble Beach in 1982).

Lastly, the putting surfaces at the U.S. Open are typically the fastest the players see all year, with perhaps the Masters or the PGA Championship the only other events matching them. The desire for quickness was first a quest for smoothness, trueness of the roll. But in recent decades, unfortunately, it has taken on a meaning of its own. 

As such, the scoring importance of every part of the game is heightened at a U.S. Open course. How much does each contribute to making it more difficult? Which factors contribute most to identifying the best golfer?

For the Record: Who Wins the U.S. Open

A review of the winners indicates that on the whole, the USGA does a good job of identifying the best golfers. (See Appendix 1 below.) The winner is most often a player of great stature. Of course, there is the case of Jack Fleck beating Ben Hogan, or Scott Simpson beating Tom Watson. But those are rare. Most often, the winner has won or will win more majors and/or many tour events.

Do the better players usually win because of the difficulty of the course or the importance of the event? The causal effect to this correlation may just be the reverse of what is thought. The fact that it identifies great champions may be more of a function that the better players try harder to win the Open, and they would do so whether the course was set up hard or easy. That is a hypothesis worth testing.

Appendix 1 (online) lists all champions for the U.S. Open since World War II, the PGA Championship since it went to stroke play and the Open Championship since 1959, when more Americans started to make the trek “across the pond.” The list of champions is quite impressive, with virtually every great player of the last half-century represented. A few outstanding players are left out. Colon Montgomery and Lee Westwood come to mind. Neither ever won a major. But almost all players of substance are in the list.

However, there are some good (read: not great) players in the list whose records don’t nearly match that of superstars like Nicklaus, Woods, Snead and Hogan. Is it possible that those very good players win more often on the “easier” setups? Is it possible that the great players win more often on the harder setups? That should be true if the idea of making a good course more difficult yields “better” champions.

To test the hypothesis, a measure of player “greatness” – a ranking of how good the winners are/were – had to be created. There are many ways to rank players. For this study, each player was assigned a number by adding 1 point for each PGA Tour win plus 5 points for each major win (each major thus providing 6 points). This was good for virtually all of the champions, although a few probably would have a higher ranking if European Tour wins were included.

Of course, this point system is arbitrary. There was no real science behind its design. There was no effort to test its “accuracy” except to scan the results and look for any inconsistencies. In that regard, it seems to be an adequate measure for this first test. Other systems were not tested.

Appendix 2 lists all the champions and their ranking under this scheme. Nicklaus with his 18 majors and 73 tour victories ranks first. Tiger Woods is second, and Sam Snead is third. In general, reviewing all the numbers, this seemed to be a fair if not perfect ranking.

Simple Statistical Analysis: Scores

Appendix 3 lists for each championship (grouped by major) the ranking of the player who won and the winning score as measured against par. It is assumed that the lower the winner’s score, the easier the course, and vice versa.

Table 1 displays several interesting figures. First, in the three majors examined (the Masters was not included because there is comparatively little year-to-year variation to the course), there is a fairly large variation in the winning score from year to year for each championship. The ranges are similar (U.S. Open, -16 to +9; PGA, -20 to +1; Open Championship, -20 to +6).

The average winning score is more interesting. The average for the U.S. Open is -2, whereas the averages for both the PGA and Open are -8. On average, the U.S. Open is “harder” by 6 strokes more than both the PGA and Open. The median and mode for the Open are even lower, indicating a slight bias toward “easiness.”

Table 1. Scores

Simple Statistical Analysis: Players

Table 2 displays the results of the player rankings for the three majors analyzed. Interestingly, the average ranking of the winner for all three was just about identical: U.S. Open, 44.3; PGA, 44.1; and the Open, 46.3. It appears that they all identify the same quality of player.

But Table 2 also displays the average player ranking for the “easier” courses (the ones whose winning score was below the median for that major) and the average for the “hardest” (the ones whose winning score was above the median). For this analysis, the U.S. Open has a higher player ranking for the winners of the easier courses than for the harder courses (49.3 vs. 42.4). The others have higher player rankings for the harder courses, with the PGA having a substantial difference and the Open a very slight (insignificant) increase.

Table 2. Player Average Rankings

What does this mean? It can mean many things depending on how you want to interpret the results. The U.S. Open results do not seem to support the assumption that the harder the course, the more likely a better player will win. However, it seems true for the other majors, particularly the PGA. But it could also mean that the course setups for the U.S. Open and the Open are more consistent from year to year and that there are other factors influencing the results. The PGA may be the outlier here because it is known that for several years, the PGA setup was not nearly as difficult as it has been recently. Or it could mean that these differences are small and insignificant in any case and the result of one or two outstanding players winning on courses on one side or the other regarding “difficulty.”

What is significant, however, is that there is no clear indication that the assumption is true.

If the purpose of setting up the course to be hard is to identify the best player, a Type I error would be when a “less than great” player wins. Of course, all of these players deserved to win because, after all, they did win. But should they have won, and did they win because the course was not difficult enough to “un-identify” them? Do those players win most often on the “easier” setups?

Table 3 displays for each major the number of winners with a player ranking below 12, with the idea that a ranking of 12 or more would indicate at least two major wins, or one major and six other tour events, or at least 12 tour events – any one of which would indicate a worthy champion. It also shows the average winning score of those players and the average winning score of the equivalent number of the highest-ranking winners.

Table 3. Lower-Ranked Players and Comparison with Top

Table 3 indicates that all three majors had about the same number of low-ranked players winning. It also indicates that their average winning score was the same or worse than those of the top-ranked players when they won a major. This might indicate that the harder courses do not identify the better players any better than the easier courses, but these differences are most likely insignificant.

But again, it does not support the assumption.

Regression Analysis

If indeed the more difficult the setup, the higher the probability of identifying the best player, then there should be some sort of statistical correlation between the winning score and player ranking. This is not an absolutely perfect concept, but it is a reasonable approach given the question at hand. If a higher winning score against par is an indication of the severity of the setup and a lower winning score an indication of an easier setup, and if it is believed that the harder the setup, the higher the probability that the best player is identified, then it could be postulated that a statistical relationship between winning score and winning player ranking should be significant at some level. The assumption would be that they are positively related. As course difficulty goes up (as measured by winning score against par), winning player ranking should go up.

A regression was run for each of the three majors under discussion (see Appendix 4 online). Absolutely no relationship was found for any of the three. The R-squared for each was essentially zero. It is possible that a higher R-squared might be obtained by taking numbers out of a random number table. The p-values for the coefficients were in the 0.44 to 0.66 range, indicating no significant relationship between winning score and winner ranking.

Again, the assumption that harder is better is not supported.

Potential Conclusions

Although this is hardly a complete analysis, some conclusions can be drawn from it, and even perhaps some recommendations.

First, from these analyses, the U.S. Open, as well as the other majors for the most part, does a good job of identifying great players (at least very good ones), indicating that course setup is generally appropriate.

However, that seems to be equally true for easier (in the context of U.S. Open setups) as well as harder course setups. It appears that an easier setup (not an easy setup) may do just as good a job as a very hard setup when it comes to identifying a great player as the U.S. Open Champion. It may be the strength of a player’s desire to win a major, in combination with their talents, more than the relative difficulty of the course, that results in identifying a champion golfer.

The U.S. Open is seemingly “harder” than the PGA and Open Championship. However, there is some evidence that it is no better or worse at identifying the best player than the other two majors. The average player ranking of the winners of all three is about the same.

As for possible recommendations, it seems advisable not to lessen the difficulty below the “easiest” of the courses that have been used but rather to allow some flexibility in setting up the course. The USGA has loosened some aspects of course difficulty by widening some fairways and providing a second cut, with no obvious negative effect on identification. Perhaps green speed should be experimented with as well. 

Postscript: Advanced Alternate Models and the Nature of Randomness

More sophisticated models are absolutely possible with better data, and those models should be created and tested before any definitive conclusions are reached. A better model would use some of the tour’s ShotLink data. More sophisticated models could be built using each player’s shot distribution (particularly their tee shots) and their innate putting skill. Simulations that place individual golfers on a specific course setup to see how the best players perform versus how the less great players perform would be a reasonable next step.

Variability is an inherent aspect of all sports. Because of its design, golf appears to suffer from variability to a greater extent than other sports. It is badly structured from a statistical standpoint. Fairways that are too wide may not separate the best from the less great. Fairways too narrow might create spurious, random results beyond a reasonable “rub of the green” result. On a narrow fairway or hard-and-fast putting surface, a nearly perfectly struck shot can have devastating consequences for a golfer, particularly at a major (e.g., Greg Norman’s final tee shot in the playoff at the 1989 Open).

Although there are many factors separating the best from the others, it can be assumed that the better players have less shot variability with regard to accuracy. At what point is the fairway width too small to reduce the random chance that the best players have no more chance of hitting the fairway than the others? What green hardness rejects good shots as well as bad ones?

What is the optimal course setup to maximize the identification of the best players? What is the most important factor to identify the best golfer? Is it length and accuracy (strength and coordination skills), short game and putting (small motion, delicate skills), or course management and nerves (mental skills)? A more sophisticated study would have to be undertaken to answer those questions.