Asking the Right Questions

A hospital invests in a deep-learning system to predict patient deterioration. A retailer builds a personalization engine to drive revenue. A financial services firm completes a proof-of-concept credit model that outperforms everything the actuarial team has been using for years. Each project invested heavily in technical work. None 
of them survived contact with the real world.

If you have sponsored, managed, or tried to scale an analytics initiative, this pattern probably feels familiar. The model works. The deployment does not. Or the deployment works, but nobody uses it – often because the solution never solved the right problem. Or it gets used for six months, quietly falls out of use, and is never mentioned again. 

Analytics failure is not primarily a technical problem. It is a lifecycle problem. And the evidence for that is mounting.

A Predictable Mistake 

Three independent research studies – a synthesis of multiple OR/MS industry surveys, RAND’s qualitative interviews on AI implementation failures, and a practitioner survey – have all converged on the same finding: analytics projects fail for recurring, predictable reasons unrelated to algorithmic sophistication.^1-3

The failure patterns are consistent: unclear or shifting project goals, unrealistic expectations set at the start, poor data quality discovered too late, analysts focused on modeling rather than decision-making, and leadership that cannot articulate what problem it is trying to solve. These are not edge cases. They are the norm. 

What makes this especially costly is that most of these failures are entirely avoidable. They occur not because the problems were intractable, but because the right questions were never 
asked at the right time. 

The INFORMS Analytics Framework^TM (IAF) shows why most failures trace back to skipped life cycle conversations. We’ll return to the framework later, but first consider three examples.

Ideal Versus Real Environments

Google’s deep-learning model for diabetic retinopathy screening was, by any technical measure, a success. Trained on tens of thousands of retinal images, it matched or exceeded specialist-level accuracy in detecting the disease. The decision to deploy it across clinics in Thailand seemed like a straightforward win.⁴

It was not. Real clinic conditions – including older imaging equipment, inconsistent lighting, variable patient positioning, and unreliable internet connectivity – produced image quality the model had never encountered in development. A significant share of scans came back “ungradable.” Nurses found themselves repeating imaging sessions, workflow times stretched, and clinic throughput fell. A system designed to improve care became an operational burden. 

Rather than, “Does the model perform well?” developers should have asked, “Does this model perform well under the actual conditions in which
it will be used?” Those are different questions, and the gap between them is where this project failed.

Defining the Goal 

A major retailer set out to build a personalization engine. Executives were enthusiastic. The technical team was capable. But the project never produced a deployable product.⁴

The root cause was simple and surprisingly common: no one in leadership could agree on what “personalization” actually meant. Different executives pushed conflicting definitions of success, scope crept in multiple directions simultaneously, and key team members were reassigned before critical work was finished. Controlled testing was impossible because success itself was never clearly defined. Users reverted to manual processes. The initiative stalled. 

This is not a story about bad data or wrong methods. It is a story about what happens when an organization starts building before it has agreed on what it is building toward. The technical team was asked to solve a problem that the business had not yet defined.

Infrastructure Is Essential

A financial services company built a machine-learning model to predict credit defaults. The proof-of-concept was compelling. The model outperformed the actuarial team’s trusted legacy models on every measure that mattered.⁴

Then the actuarial team pushed back. IT imposed security protocols that blocked the analytics team’s shared environment. The CIO who had championed the project left the organization. Without executive cover and with no shared infrastructure, the project stalled at the proof-of-concept stage. The better model sat unused while the inferior legacy system continued making decisions. 

Although technical quality is a necessary condition for analytics success, it is not a sufficient one. An organization that does not align its stakeholders, understand and secure its infrastructure, and establish durable sponsorship before deployment will lose projects even when the underlying work is excellent.

A Common Thread

Across these cases and many others, the failures trace back to the same set of skipped conversations. Was the problem or question the correct one and agreed upon? Was the problem amenable to an analytics solution? Were the right stakeholders aligned before work began? Was the data validated not just for quality, but also for what it would look like in a live operating environment? Was the workflow ready to absorb the output?

These are not questions most organizations ask systematically. They get asked when something breaks – and it’s too late. 

The IAF formalizes these questions so they become part of every analytics lifecycle, not an afterthought. The IAF structures analytics work across seven domains covering the full life cycle: from framing the business problem to defining success through data readiness, model development, deployment, and long-term performance management. It does not promise that every project will succeed. What it does is force the right conversations earlier, when course corrections are still inexpensive and achievable.⁵ 

Had the Thai clinic deployment team validated model performance against real clinic conditions before launch, the operational failure would likely have been caught in testing. Had the retailer’s leadership aligned on a clear, shared definition of personalization before committing resources, the scope creep that killed the project would have had nowhere to take root. Had the financial services firm secured IT infrastructure and stakeholder buy-in as part of the deployment plan, the credit model might be running today.

Analytics as an Organizational Capability 

If you sponsor or oversee analytics work, the single most valuable thing you can do is insist on rigor before the modeling begins, not afterward. That means demanding a clear, written problem statement before any data is pulled. It means requiring an explicit assessment of whether the problem is solvable with available data and infrastructure. It means ensuring that deployment conditions, not just testing conditions, are part of the validation plan from the start.

It also means treating analytics as an organizational capability rather than a sequence of one-off projects. The organizations that get lasting value from analytics are the ones that build repeatable processes for asking the hard questions early, tracking solution performance over time, and maintaining the institutional knowledge that makes the next project faster and smarter than the last. 

The IAF provides a shared vocabulary and a systematic approach for the life cycle decisions that determine whether analytics work reaches the people it is meant to serve. It is not a guarantee of success, and it is not a substitute for strong leadership, adequate resourcing, or sound project management. Rather, it is a discipline, a decision protocol that helps analytics programs build on what they learn instead of repeating the same first steps. 

Most analytics failures are not surprises. The signals are there early. The organizations that catch them are the ones that have built a discipline around monitoring signals in advance. 

Laying the Foundation

Ask the IAF readiness questions in Figure 1 before green-lighting any analytics initiative. Each question maps to one of the seven IAF domains and represents a category of failure that organizations consistently skip past, often until it’s too late. 

The IAF does not eliminate risk. It surfaces it when something can still be done about it.

References  

Aho, T., Kilamo, T., Lwakatare, L., et al., 2021, “Managing and Composing Teams in Data Science: An Empirical Study,” 2021 IEEE International Conference on Big Data, https://ieeexplore.ieee.org/document/9671737.  

Henrion, M., 2019, “Why Most Big Data Analytics Projects Fail,” OR/MS Today, https://pubsonline.informs.org/do/10.1287/orms.2019.06.08/full/.  

Ryseff, J., De Bruhl, B. F., Newberry, S. J., 2024, “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed,” RAND Research Reports, https://www.rand.org/pubs/research_reports/RRA2680-1.html.  

Gray, D. A., Shellshear, E., 2025, Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, Without the Hype, Chapman & Hall, https://www.routledge.com/Why-Data-Science-Projects-Fail/Gray-Shellshear/p/book/9781032660301.  

INFORMS Analytics Framework, 2026, https://www.informs.org/Professional-Development/INFORMS-Analytics-Framework/Explore.