Unrealistic Expectations

“Under Promise, Over Deliver” – Tom Peters, TPG Communications (1987)

One of the critically important lessons that I personally learned the hard way earlier in my career involved setting (un)realistic expectations.

There are two primary domains for expectation setting that plague IT and data science professionals alike:

1. Project-related variables, namely scope, timing, resources and budget.
2. Business value and economic impact.

Numerous books have been written on Agile Scrum and Kanban estimation, and much of estimation is an art and a science learned over time from lots of practical experience. My only recommendation here is to balance conservatism and stretch goals. It is always better to be a bit early than late, relative to the promised deadline.

In the second domain of business value and economic impact, balancing conservatism and stretch goals is also advisable. My favorite graphic to illustrate this point is shown below.

We establish an expectation-setting continuum.

At the far right of the spectrum is where we set the bar for benefits too low, sailing over the bar too easily and blowing away our target. This approach, known as “Sandbagging” tends to lose a customer’s confidence because they perceive the data scientist as not being aggressive enough in targeting potential business value.

At the left end of the spectrum, we set the bar for benefits too high and fail to deliver the promised business value. This approach, known as “Overcommitting” – or an “epic fail,” as millennials might say – can get you into really big trouble with customers (and their bosses/executives) because they were expecting to deliver a monumental economic impact and came up way, way short (e.g., expecting fireworks but got sparklers).

An example of sandbagging versus overcommitting would be achieving a $100 million verifiable cost avoidance but promising $10 million and $1 billion, respectively. With sandbagging, you blow your target away times 10, and with overcommitting, you miss the mark times 10. Both are bad, but overcommitting can be politically irredeemable and career jeopardizing.

In business school, MBAs (of which I am also one) are taught to always analyze (at least) three outcome cases in any analysis or projection modeling scenario:

1. Best case (mostly everything goes right).
2. Worst case (mostly everything goes wrong).
3. Expected or average case (some things go right, some things go wrong, and it balances out).

The process of estimating benefits is similar, and the last case is in the middle of the expectation-setting spectrum, in which we try to balance our (and the customer’s) optimism and pessimism and use an expected or average case to set a target we can hit (or even exceed) without being too far off in either direction. I call this approach the “target zone.” To continue the example above, we would estimate, say, an $80 million benefit and deliver $100 million. The upper end of the target zone, and just beyond, is sometimes referred to as BHAGs, or Big Hairy Audacious Goals (see “Built to Last: Successful Habits of Visionary Companies” for elaboration on BHAGs). You may have heard BHAGs referred to as “stretch goals.”

Finally, to finish the above example, we would set a target zone aim of an $80 million benefit and a BHAG of $120 million and deliver a $100 million benefit. Regardless of whether you achieve the BHAG, it is better to aim a bit higher and push for the larger opportunity. Even if you don’t achieve the BHAG, you won’t fall as far short by overcommitting as wildly as with the $1 billion benefit.

Clearly, the examples above are contrived because we have the benefit of hindsight, or perfect information, and we achieved a healthy business value outcome of $100 million. That does not always happen in reality. Sometimes the benefit is $0 or something close. Sometimes we get lucky and hit the jackpot.

Setting Business Value Benefits

I’ll provide some key learnings on setting business value benefits and economic impact targets before a project commences.

1. The size of the business, in scope and scale, measured in terms of sales, revenue, costs, assets, labor force and profits, greatly matters in how much business value can realistically be achieved in general and in any one project within the business.

Some of the greatest achievements in the history of data science (operations research, analytics) at Fortune 50 companies were nine-figure annualized business value improvements:

• American Airlines’ yield management system (DINAMO) was validated to have generated $1.4 billion in incremental revenue over a three-year period (and was expected to deliver ~$500 million in incremental revenue annually in the future) with a similar fleet and airline schedule (see “Yield Management at American Airlines”).
• UPS’ On-Road Integrated Optimization and Navigation System (ORION) avoids $300 million to $400 million annually in fuel and driver costs (see “Meet ORION, Software That Will Save UPS Millions By Improving Drivers’ Routes”).

Although there have been many examples of even larger annual business value benefits of data science and similar fields, the 9- to 10-figure dollar range provides a reasonable upper bound on the largest possible practical expectations.

2. The best place to start is with the firm’s financial statements to understand financial performance to estimate benefits, and then examine a given department’s contributions to the firm’s financial results (Marketing, Manufacturing, etc.).

Key areas of the business for economic opportunity include:

• Labor, e.g., through AI-based robotics in warehouses, factories and, in the future, driverless vehicles, including cars and large trucks, as well as optimization-based labor planning systems.
• Inventory, e.g., better matching of product demand to supply to balance shortage and holding costs.
• Asset (including facility) allocation and utilization, e.g., aircraft (hangars), railroad engines and rolling stock, and tractor trailers.
• Manufacturing, e.g., product mix, process controls, statistical quality control.
• Pricing, e.g., Walmart was a 2020 INFORMS Franz Edelman Award finalist with the predictive (demand forecasting)-prescriptive markdowns optimization solution that balanced discounting goods too much or too little, too early or too late, to maximize sales revenue.
• Yield or revenue management, e.g., originating in airlines and now utilized in hotels, cruise lines, rental car companies and even self-storage facilities.

3. When estimating benefits, first calculate the maximum potential benefit (at 100% realization), and then perform a rigorous analysis based on the firm’s actual economic data to evaluate how much business value and economic impact is realistically possible to achieve with data science.

The latter figure may very well only be 10%-25%, and you may decide to set a target zone goal at 5%-10% of the maximum to avoid sandbagging or overcommitting. Rarely, if ever, will you achieve the maximum benefit, but multiplying 10% times a very, very big cost or revenue dollar figure can still be a significant number in itself.

4. Look for the largest potential business value opportunities in your company where there is significant room for improvement in economic efficiency to see how data science can provide the greatest leverage. Look for complex, large economic impact problems that are currently being solved essentially manually in Excel with rules of thumb or simple heuristics that do not capture the fullness of the problem or solution opportunity. 

In the airline industry (American, Delta, et al.), the largest opportunities were found in seat inventory pricing and yield management, which directly impacted revenue, followed closely by network planning/flight scheduling and flight/cabin crew scheduling and fuel inventory management, which are an airline’s two largest cost categories, followed by spare parts inventory and aircraft maintenance.

In the package delivery industry (UPS), the largest opportunities were found in optimizing the operations of their fleet of 55,000 delivery trucks and drivers (not making left-hand turns because you burn more fuel waiting to turn!).

Setting realistic business value and economic impact targets and expectations will depend on how well you understand the economics, operations and financials of your company and then how rigorously you analyze the impact that data science can potentially have by utilizing the framework of target zone and BHAGs from the expectation-setting continuum.