Assessing the analysts

As the appreciation for analytics spreads, more and more decision-makers confront an urgent question: How do they recognize whether analytical work is done correctly? The Institute for Operations Research and the Management Sciences (INFORMS), among other professional societies, is moving toward a certification program, in part in response to these decision-makers’ needs. Certification, however, is all too often implemented with a heavy emphasis on technical skills because that is what is easy to measure. Even if “soft” skills are included, the focus is still on capabilities, not on actual accomplishment in specific situations. Therefore, certification is unlikely to meet the client’s need to assess the quality of the O.R. products or approaches. Another activity is closer to the point: training the clients of analytics to assess analytical products and performance.

When technical people do badly, it is rarely because their technical skills are deficient relative to what would be expected given their education and experience. Rather, they more often misunderstand the problem, fail to communicate clearly with the client, fail to learn and adapt as the situation changes, or reach beyond what they know well without conveying their own increased uncertainty. Knowing that an analyst is generally competent is not sufficient to give the client reasonable assurance that the analyst’s recommended solution is good. Wise clients do not micromanage technical tasks, but they do ask insightful questions about assumptions, alternative approaches and known risks in the chosen solution. Helping clients get wiser would benefit both those clients and the analytics profession.

The August 2011 issue of OR/MS Today (the membership magazine of INFORMS) includes a description of dramatic improvement in hospital emergency departments’ functioning via the application of some fairly simple analytics tools and techniques [1]. David Eitel, an emergency department physician, learned enough to come up with the key ideas in a couple of courses in an executive MBA program. Even more striking, however, is his statement that before he took those MBA courses, he had no idea of what operations research/management science/analytics was or what it could do, let alone whom to ask if he wanted to find out. Now he wants to get other health care managers as excited as he is about the potential of analytics, but clearly the first hurdle is teaching them what that potential is and how to engage it.

These managers’ reticence about using analytic methods, when they do learn about them, is based in part on a well-justified concern about how to know whether these methods really work. Douglas W. Hubbard, in an OR/MS Today article in 2009 referencing his then just released book about risk management, summarized a study he conducted in several dozen U. S. companies [2]. He found that few quantitative analysts had applied quantitative measures to evaluate how well their forecasts and recommendations had come out.

It would be useful for the analytics profession to develop guidance for decision-makers about how to assess the quality of analytical work. Venues in which to apply this guidance include hiring and performance evaluation of employees, selection and direction of contractors, and appreciation of the strengths and weaknesses of competitor organizations. Some issues a good assessment approach would address include:

• What assumptions are involved in the method you chose?
  
  
• What alternative methods did you consider, what assumptions do they require, and on what basis did you choose?
  
  
• How available and how good are the data your method requires?
  
  
• Are you sure you have the problem stated as clearly, accurately and thoroughly as necessary? Have you revised your problem statement since starting the task? If so, what had you learned that motivated the change?
  
  
• What have you done or plan to do to familiarize yourself with the problem and its context, and with the subject matter field in which it arises? (Prefer analysts who want to see the problem first-hand – best by immersion in it, if possible.)
  
  
• In your experience, how often are revisions of the problem statement required as you learn more about the situation? (A response of “never” should result in instant disqualification.)
  
  
• Who else would know about this, and what challenges might they raise? How would you respond to those challenges? (This is best done, if possible, by bringing in strong challengers as “red team” peer reviewers, rather than simply conjecturing as to what issues they would identify. Even conjecture, however, is better than not even thinking about these subjects.)
  
  
• What criteria do you use to decide when a less than certain answer is close enough? (“I always try to get all the way to the provable optimum” should also result in disqualification.)
  
  
• If we had more time and resources, what else would you want to know about this problem? (Highest marks here go to the analyst who has done serious, systematic analysis of what conjectural new information, if it appeared, would totally thwart the recommended approach, and of how well we know how likely such an appearance is.)
  
  
• How would you assess, over time, how well the adopted approach is working?
  
  
• What indications and warnings would you look for that might lead you to reconsider what you recommended?
  
  
• How easy is your solution method for others to use?
  
  
• How easy is your solution method for others to modify?
  
  
• How do you intend to document what you did so that others can use it, adapt it, modify it or take issue with it in the future?

This list is far from exhaustive, but it definitely captures many of the elements most experienced technical managers consider essential, in this reporter/analyst’s experience. It is certainly sufficient to form the basis for a short course for current and would-be clients and managers of analytics. Such a short course would most likely generate some lively discussions that would, in turn, inform better analytics practice.

REFERENCES

1. David Eitel and Douglas A. Samuelson, “O.R. in the ER: How O.R. Can Help in the Emergency Care Crisis,” OR/MS Today, August 2011.
2. Douglas W. Hubbard and Douglas A. Samuelson, “Modeling Without Measurement: How the Decision Analysis Culture’s Lack of Empiricism Harms its Effectiveness,” OR/MS Today, October 2009.