Practical Optimization

Rating:
Summary: Was good in its day
Review: After some initial use, this book sat unused in my library for many years. We were reacquainted recently when I needed help in solving a nonlinear optimization problem with some tricky aspects.

Rather than being a cookbook of optimization algorithms, it is a thoughtful text about optimization problems and the algorithms used to solve them written by experienced researchers who contributed algorithms for the renowned Numerical Algorithms Group (NAG). It leads one through the classification tree of algorithms (much as you'd do at, e.g., NIST's Guide to Available Mathematical Software), but with chapters and sections devoted to each category or subcategory so you can fully understand the approach and its behavior in detail. This book therefore complements the more typical algorithm- and package- (e.g., Matlab) oriented treatments.

Strengths: a) Good comparisons of similar methods with advice on when to use each. b) Nice discussions, brought alive by many graphs and simple but subtle example problems to illustrate each algorithm and its idiosyncracies.

Weaknesses: a) Sections don't stand alone. I found myself flipping back to early chapters where certain formalisms are introduced. b) The book is dated. The 70's approaches to solving a constrained problem by using penalty functions with a series of unconstrained optimizations have largely been supplanted by new methods based on the Kuhn-Tucker equations. c) There is no coverage, obviously, of other important modern techniques (simulated annealing, genetic algorithms, neural nets) developed after the book was written in 1981.

In summary: It was good in its day, and may still be helpful in understanding the classic algorithms and choosing the best. You'll need to look elsewhere, however, for a modern treatment.

Rating:
Summary: Old, but still helpful
Review: This is a clear, useable introduction to basic techniques for numerical optimization. It addresses the classic kinds of optimization targets: differentiable (mostly) and unimodal (mostly), with or without constraints of various kinds.

Maybe that sounds a little naive and optimistic by today's standards. It still describes a large class of useful problems, though, and may help with end-stage refinement of local optima found other ways. Even genetic optimization algorithms can benefit from classical techniques - a mutation operator may actively seek local improvement instead of relying purely on random perturbation.

Gill covers a variety of techniques, using the first or second derivatives or discrete approximations. He also presents a simplex technique, not to be confused with the simplexes used in linear optimizations. Throughout, he puts strong emphasis on numerical stability and on the relative strengths and weaknesses of each approach, all the way back to the linear systems that approximate each step of the analysis.

It's not a recent book, but that is part of its value. Many of its techniques are almost lost. They have been included by specialists into canned analysis tools, hidden from sight, and forgotten even by the people who use them. I often need to go back to these basics, and very few more recent books are as clear or complete, over this book's range of topics.

Gill has written clearly enough for a careful programmer to implement the algorithms. There is not much here for theoreticians or cut&paste hackers, but it's a useful reference for someone willing to put the time into understanding.