How scipy.optimize.minimize Chooses an Algorithm

In Python’s SciPy library, the scipy.optimize.minimize() function automatically selects an optimization solver when the method parameter is not explicitly defined. This article explains the decision tree SciPy uses to choose between algorithms, why gradient-based solvers like BFGS are favored over derivative-free methods like Nelder-Mead by default, and how to know when manual solver selection is necessary.

The Default Selection Hierarchy

When you execute scipy.optimize.minimize() with method=None, SciPy does not inspect the mathematical properties of your objective function or check whether you supplied an analytical gradient (jac). Instead, it relies entirely on the presence of problem constraints and parameter bounds.

The internal selection logic follows a simple priority sequence:

1. If constraints are provided (constraints parameter is non-empty):
   SciPy selects SLSQP (Sequential Least Squares Programming), which supports both equality and inequality constraints, as well as bounds.
2. If bounds are provided (bounds is not None), but no constraints:
   SciPy selects L-BFGS-B (Limited-memory BFGS with Bound constraints), an algorithm designed to handle parameter ranges efficiently.
3. If neither bounds nor constraints are provided:
   SciPy selects BFGS (Broyden–Fletcher–Goldfarb–Shanno).

Why BFGS is Chosen Over Nelder-Mead by Default

A common misconception is that SciPy will fall back to Nelder-Mead if the user does not supply an analytical Jacobian (jac=None). However, Nelder-Mead is never selected by default in modern versions of SciPy.

When BFGS is invoked without an explicit gradient function, SciPy automatically approximates the Jacobian numerically using 2-point finite differences. BFGS is preferred as the unconstrained default for several reasons:

• Convergence Rate: BFGS exhibits superlinear convergence for smooth problems, making it significantly faster than Nelder-Mead in finding high-precision minima.
• Dimensionality Scaling: Nelder-Mead uses a simplex heuristic that degrades rapidly as the number of dimensions increases (often failing or stagnating on problems with more than 10 to 20 variables). BFGS scales far better to moderate- and high-dimensional spaces.
• Reliability on Smooth Objectives: For continuous and differentiable loss functions, finite-difference gradient approximations combined with quasi-Newton updates are consistently more reliable than heuristic direct-search methods.

When to Manually Specify Nelder-Mead

Because scipy.optimize.minimize() defaults to gradient-approximating algorithms, you must explicitly pass method='Nelder-Mead' when working with problems unsuitable for BFGS:

• Discontinuous or Noisy Objectives: If the objective function includes stochastic noise or step discontinuities, finite-difference approximations will yield misleading gradient vectors, causing BFGS to fail. Nelder-Mead evaluates only function values and handles noisy landscapes more effectively.
• Non-Differentiable Functions: Absolute-value penalties, min/max functions, or discrete simulations where true gradients do not exist.
• Very Low-Dimensional Problems: For simple 1D or 2D heuristic search problems where numerical stability is less sensitive.

Summary

scipy.optimize.minimize() determines its solver strictly by the presence of constraints and bounds—defaulting to SLSQP for general constraints, L-BFGS-B for bounded parameters, and BFGS for unconstrained problems. Because Nelder-Mead is a direct-search algorithm intended for non-differentiable or noisy functions, it is never chosen automatically and must always be requested explicitly.