Optimization engagement baseline

An earlier article [1] showed how a simple measure, Time To Payback (TTP), can be a major help in the definition of an optimization development project. One of the goals of any iteration is to deliver (financially) rewarding results to the client in line with the corresponding investment. The TTP rule of thumb: a specific investment should be less than six months for the project to be safely successful. This, or at least this way of thinking, should apply to every iteration.

Two phases are particularly sensitive to this rule: the first phase, where set-up costs for the overall engagement drive the TTP to a higher level than if these costs were spread over the entire engagement; and the last phase, where marginal benefits (those gained relative to those obtained at the end of the previous iteration) may be lower than what would deliver such a TTP but are nevertheless acceptable in view of the overall delivery.

This article focuses on the first phase, which is the gateway to a successful engagement and which we’ll call “baseline” development. At the beginning of many large projects, this approach has often been called “demonstrator,” “proof of concept” or “prototype.” The demonstrators typically aim at proving the technology or the approach, rather than gaining immediate (financial) benefits for the client, and can badly miss the mark.

Useful directions for the first phase, both in terms of issues to address and of results to obtain, can make this phase reap up to one-third of the overall benefits of an optimization engagement.

Common issues:

• most data needs cleansing or at least “getting used to”;
• most specifications need updates;
• the engagement benefits greatly from having end-users (e.g. planners) involved and onboard from the start; and
• many costs and benefits models need tuning if not outright definition.

Expected results:

• validate the graphical user interface (GUI);
• do at least as well as the existing system, in an automatic manner;
• define an acceptable optimization model;
• get a facility to try things out; and
• get an effective testing support.

Issues and Inputs

The main issues surrounding an optimization engagement, beyond the optimization modeling and tuning, are very standard IT aspects, such as:

• integration;
• incomplete definition of the optimization environment, typically in terms of data to be used and results provided; and
• constraints, costs and benefits pertaining to the application.

The earlier one can fully validate these elements, the better off the engagement is going to be. Waiting for the availability of optimization results to validate these elements with end-users’ expectations delays this validation as well as the associated changes until late in the project, which, in most cases, is not necessary since most if not all of these elements can be collected in a constructive way during the baseline development, without much, if any, optimization involved.

Data Handling

Most data used for the optimization engagement already exists and has been used for a different purpose. The quality of the optimization application results depends on the quality of the data delivered to the optimization application as seen from this application, not as seen from the earlier ones using this same data.

One needs to have the tools and means in place to either cleanse the incoming data or be able to “live with it,” i.e. have the optimization model be robust enough to handle such data in a way similar to what the planners do in their daily jobs [2]. Planners are used to doing this manually and are the best source of processes and rules for this handling of data.

Experience shows that if data handling is delayed and only done simultaneously with the optimization tuning, the two issues build on each other and yield an almost intractable set of problems. As there is no development or technical requirement that forces data evaluation to wait until later in the engagement, it can and should be done quite early, with the full involvement and support of the planners.

Specifications Updates

Optimization specifications are easy to enunciate but often change when faced with actual results. The earlier these specifications can be stabilized, the more effort can be spent tuning the optimization, and the better the results can be.

In a way similar to data, elements that are considered to be obvious to the planner/scheduler may not be so to the optimization modeler, and may be missed entirely until the tuning phase of the application, which is a very costly approach both in terms of efforts and delays.

End-Users Involvement

There are at least two other reasons – one technical, one organizational – to involve the planners/schedulers as early as possible.

The technical reason is that planners and schedulers in place are usually very good at what they have been doing, and that any optimization application benefits from their accumulated knowledge. Indeed, their (often unformulated) optimization model is a very good starting point for the work to be done.

The organizational reason is less immediately visible. Success or failure of the new optimization application depends in large part on the acceptance of this application by the existing crew of planners and schedulers. If feeling threatened, they can, in most cases, show that at least part of the new application doesn’t do as well as what they can do. It is essential to obtain their support as early as possible in the development of the application. One constructive way to do so is to show how the application supports them in their work.

Costs and Benefits Tuning

Just like specifications firm up when faced with the optimization results, the definition of the costs and benefits functions gain by being confronted with practical results.

Oftentimes the cost or benefit element is chosen for simplification’s sake as an indirect measure of what the user actually wants to optimize and is not well defined, or often even understood, as long as it does its job correctly. Adding an effective optimization tool shows all the potential blemishes. The earlier these are corrected, the better the optimization engine will be able to do its job.

Results

Just as many of the inputs and constraints to the optimization engagement do not require the optimization engine results in order to be validated, many of the outputs and support results do not need the optimization either. In particular, the representation of results to the planners and schedulers is independent of the detailed optimization results, even if the structure may depend on the underlying optimization model. Furthermore, the earlier the planners can be brought in to validate the main assumptions of the application, the more robust the results are going to be. Planners and schedulers are best positioned to validate the results of the optimization effort; their early access to the representation of these results greatly eases the testing and tuning effort.

Graphical User Interfaces (GUI)

Humans are very visual in their daily interactions. This seems to be especially true of planners and schedulers who have been used to modeling their work with visual approaches and tools. Using interactive graphics in order to support their work makes sense. Many simple planning and scheduling applications do just that, and do it quite well, without any optimization.
Planners and schedulers are used to GUIs and have become good at spotting issues when faced with a familiar representation of their system.

An optimization engagement gains by supporting a familiar representation of the elements to optimize. This has several advantages, some mostly linked to a better understanding of the existing system (detailed in the next section) and some linked to not ruffling the planners’ feathers, while keeping them in a familiar environment in which they can effectively and constructively explain what works and what does not. This representation doesn’t require any optimization engine but benefits strongly from:

• optimization constraints propagation that delivers “legal” if not optimized solutions, and
• an interactive GUI that allows planners and schedulers to easily propose a solution while the calculation of the corresponding KPIs is done in parallel.

Existing System

The best reference for all participants is the existing system. It makes sense to closely replicate this system in the future environment, for a number of human and technical reasons. Most of the human reasons have been mentioned above and include intellectual comfort and support. The technical ones include:

• access to the real data. As mentioned above, this data is usually not perfect, yet the planners know how to handle it effectively. Accessing it reveals these imperfections and the associated access patterns and avoids difficult validation efforts during the tuning phase of the optimization engine;
• validation of KPIs and cost functions;
• discovery and validation of effective heuristics used, consciously or not, by the planners; and
• tuning of the GUI and of the associated reporting on real scenarios.

The added tools and reporting patterns associated with the new system should already bring a significant added value to the planners and operations. This typically represents 15 percent to 30 percent of the overall benefits brought in by the full application.

Optimization Model

The baseline aims at the identification and display of a good optimization model with a focus on “legal” results, which respect the optimization constraints expressed by the customer. It also displays the calculated costs and benefits as expressed by the customer.

Notice that this “intelligent GUI” does not attempt to optimize the results. It makes sure that the plans entered manually are “legal” and that the corresponding costs and benefits are accurately displayed.

Experience shows that this is of great value to all parties, both in the short- and in the long-term:

Planners and schedulers receive effective support to create plans and schedules, with the ancillary tasks properly taken care of, and the system making sure that they generate “legal” solutions. This often reduces the planning and scheduling effort by up to 50 percent.

This approach yields an accurate set of optimization definitions, before the development team starts the optimization tuning effort per se for, as it validates:

• the constraints to be used for this application (Indeed, the end-users will promptly identify if a schedule is valid or not, thus leading to an accurate and agreed upon set of constraints to be used for optimization.);
• the costs and benefits functions as well as the KPIs, experienced planners “know” what works and what doesn’t and can explain the discrepancies they see between the “legal” solution as produced by the baseline and what it should be;
• the data usage, as this GUI is based on “real” data; and finally
• the GUI to be used in the final application and trains the users on it.

Independent of the benefits that this approach delivers to the overall project, this development is a central and fairly difficult one, as it has to:

• define a sufficient optimization model for the system; and
• represent this optimization model part of the overall system, and not necessarily all the details of the current system, in a way that makes sense and is useful to the end-users.

It is often difficult to make the distinction between the optimization model and the overall system. This distinction effort is worth it, as it should ensure that the optimization model covers enough of the process to be of use since:

• typical users and junior modelers will want more details more in line with the physical environment, while
• seasoned optimization experts will focus on having a model that is as concise as possible, while staying representative enough to be of real long-term use to the planners and schedulers.

Delivery of a sufficient optimization model that underlies the intelligent GUI that is made available to the planners benefits all participants in this type of engagement.

Facility to Try Things Out

The baseline approach allows planners and schedulers to work with the future system, albeit with partial functionalities (typically with no optimization support or just equivalent to the current one), while validating and identifying the: data inputs, typical results, KPIs and costs, GUI (if available), and the most successful heuristics and the reasoning behind those.

In particular, the “intelligent GUI” helps planners define legal plans and schedules, without having them spend inordinate amounts of time making sure that the various constraints and requirements are respected. It allows them to toy with various valid schedules and compare their outcomes instead of focusing on the delivery of a valid schedule, with as good an outcome as possible. This is all what typically have time to do when unassisted.

A frequent issue in optimization engagements is the requirement for explanations for the results obtained. In many cases the optimized result is not the one expected as it optimizes the overall schedule, leading in some instances to locally suboptimal results. Explanations for these results can be complex, especially if the users are unfamiliar with the overall system being optimized. The baseline approach allows the users to compare various scenarios and provides the opportunity to show that a better global scenario can be locally sub-optimal, thus making the explanation effort much more intuitive.

This baseline approach is the first major step on the way to the full optimization application, as once the baseline is validated by all parties, the development of the optimization can take place with significantly reduced risks because:

• the data are validated,
• the interfaces and user habits are taken into account,
• the KPIs and costs are agreed upon, and
• existing heuristics can be reused if necessary.

Testing Support

Testing is different from “trying things out” in that it aims at the validation of the application with respect to the specifications and various requirements. Trying things out attempts to find what can be obtained and understood from using the application. Testing is especially tricky in an optimization engagement as not only should the functionalities of the application follow the requirements, but also this application has to deliver “optimal” or at least “optimized” results. A typical application, as shown in the previous TTP paper, is expected to bring in overall returns of twice the total development costs every year it runs.

Where the baseline approach is applicable with an “intelligent GUI” as described above, planners and schedulers are very effective at delivering a visual validation of proposed plans and schedules, and also quite often at the identification of issues and potential improvements.

Also, the application developers learn many of the “tricks of the trade” from the planners, and can visually spot planning and scheduling inconsistencies and errors long before those would be found with numerical results.

Note that this visual approach to testing should in no way remove the obligation to have automated testing, as this automation is going to be the only safe approach to regression testing and having an application where all the results are consistently validated during the development and tests. Many results of an optimization engagement can be tested and validated early on, especially when a baseline approach with intelligent GUI is used. At the end of this phase, all the conditions are ready to focus on the automation of the optimization aspects of the engagement.

Conclusion

The TTP approach leads to the definition of a baseline with “intelligent GUI” as the first phase of an optimization engagement. This baseline offers significant returns to the planning and scheduling departments. In some cases this can bring in enough returns that some software and services companies focus their offering on this functionality only.

The proposed approach outlined in this article is different; it has this baseline as a first, yet rewarding, step on the way to a full optimization engagement. It both:

• delivers significant benefits (albeit potentially closer in some cases to the 12 months line on the TTP than to the six months recommended for the full application); and
  
  
• opens the door to a safe and effective optimization engagement where most of the usual issues have earlier been tackled and solved in an environment that is easy to manage.

Acknowledgments

The author acknowledges IBM colleagues Filippo Focacci, who had the original idea of TTP; Jean Pommier, who steered the development and application of ISIS; and Dan Vandenbrink, who introduced the concept of baseline into Logic Tools.

References

1. Maurice Schlumberger, “Time to Payback,” Analytics, January/February 2010, pp. 23-26.
2. William Pulleyblank, “Twenty-first Century Optimization Challenges,” DIALOG 09, Orlando, Fla., February 2009.