There's Always 'Model Risk'

I would like to point out a few misconceptions from the interview with Stephen Baker in the Winter 2008 issue of Analytics. [http://viewer.zmags.com/publication/d10da139#/d10da139/15]. I was struck by Mr. Baker’s comment that “...if Takriti can do that with IBM employees [model consultants’ skills for optimal deployment], other[s] ... can do the same thing with other groups of people.” The introduction to the interview went on to say that this was a concept “most famously pioneered by companies such as Amazon and Netflix.”

It turns out others have been modeling complex human behaviors since the 1950s. The true pioneers in this area were Fair, Isaac and Company. They have been building models for the credit industry since 1958. Marketing analytics firms, such as National Decision Systems, now part of Claritas, have been building data based models of human response, for marketing, since the early 1970s.

Sophisticated modeling has been fairly common and increasingly complex since the 1990s when HNC, now part of Fair, Isaac, started using neural network techniques to identify fraud in credit card use. Many factors have made this kind of analysis and modeling more prevalent in this century:

• larger data bases from companies such as Axciom and the credit bureaus;
  
  
• expanded, cheap data storage;
  
  
• complex algorithms in commercial software with easy to use interfaces;
  
  
• computing capacity on small, inexpensive platforms;
  
  
• more acceptance of analytical solutions in the board room; and
  
  
• better training in colleges and universities, including expanded programs in statistics, operations research, econometrics and industrial engineering focused on modeling.

Mr. Baker made an interesting comment on modeling and the credit crises, regarding garbage in, garbage out (GIGO), which I think was a tad off base. In my opinion, one of the biggest problems with sophisticated models in a business environment is that management hasn’t felt the need to really understand and appreciate the complexity/ details of these models and the data upon which they were built.

Any experienced credit manager would have recognized that the mortgage data from the late 1990s through the mid-2000s were an aberration. During this time, people who could not handle their mortgage simply refinanced or sold because prices were going up; but, when prices stabilized or fell, things were different. Reselling to pay off the loan was no longer an option. This resulted in increasing foreclosures. Anyone who is at all familiar with the mortgage market has seen similar cycles in selected parts of the country in the ’60s, ’70s, ’80s and early ’90s. Building models on data from 1995-2005 would not capture the down part of the cycle.

That is not an example of GIGO but an example of an inappropriate sample. The data was not garbage; it just did not reflect long-term reality. This is called truncated data as the data does not have the appropriate cases to reflect known realities. Management should have been aware of this and made sure the modeler’s used the appropriate data samples to reflect the true market place. Another option is to have judgmental compensations within the model for the downturn effect (at FICO this is called “model engineering”).

MODELING BEST PRACTICES

Mr. Baker could have taken the opportunity to reflect on succinct ways to improve the modeling process. One of the basic management tenets should be that any model that can not be explained and understood by management should not be implemented. Senior management should know the following basic facts about the model:

• What are the strongest variables?
  
  
• What are the directional relationships?
  
  
• What transformations are used including capping, flooring and binning.
  
  
• All variables must be capped and floored to prevent unwarranted extrapolation, such as extrapolating increasing housing prices.

Unfortunately this is not the case.

After all, the company is betting its future on these models being correct and accurate. If a modeler cannot reduce models to these simple concepts then they should never be approved.

I do agree with Mr. Baker regarding the use of automation: “… we can do 10 times the volume and make 10 times the money if we automate.” But, it is more like the processing is increased by factors of thousands. I would add that automation also gives you consistency of decisions, both right and wrong. If a mistake is made, it is made a thousand times as fast resulting in a thousand times the losses.

As the Office of the Comptroller of the Currency has stated in OCC Bulletin 2000-16, there is always “model risk.” Any company that does not do its best to mitigate model risk (sample validation, implementation validation, model performance validation, and ongoing monitoring/tracking) is out of compliance.