An Evaluation of Substitute Methods for Derivatives in Unconstrained Optimization

Under what circumstances can finite difference approximations serve as substitutes for analytical derivatives in unconstrained optimization? Are such techniques as good as using derivative-free methods? We investigated these two questions via both theoretical analysis and the solution of 15 test problems on the computer. It was discovered as expected that the error in the derivative substitutes influences algorithm performance only near termination, and that simple difference formulas are more efficient than more complex ones if the step size is selected properly. Among all the algorithms tested, the Davidon algorithm was the best both for the class using analytical derivatives and for the class with finite difference substitutes. Derivative-free algorithms were not as efficient.