5.4.10 · D5Scientific Computing (Python)
Question bank — scipy.optimize — minimize, fsolve, curve_fit, linprog
Before we start, three plain-word reminders so every term below is earned:
True or false — justify
TF1. minimize always returns the global minimum.
False — it returns a local minimum near the starting guess
x0; only for a convex objective (see Convex Optimization) is that local point guaranteed global.TF2. linprog needs a starting guess just like minimize does.
False — a linear program's optimum is always a vertex of the feasible region, so the simplex or interior-point solver walks the corners deterministically; no seed is required.
TF3. fsolve finds every root of the system.
False — it converges to at most one root, the one whose basin contains
x0; other roots need other starting points.TF4. Squaring the residuals in curve_fit is just a convenient trick with no deeper meaning.
False — squaring both prevents positive and negative errors from cancelling and yields the maximum-likelihood estimate when the noise is Gaussian, so it is principled, not arbitrary.
TF5. Minimizing and finding a root of are two unrelated tasks.
False — at any smooth minimum the gradient is zero, so minimizing is root-finding on ; that is why Newton's method underlies both APIs.
TF6. For a linear model, curve_fit still needs many iterations to converge.
False in spirit — for a linear model the normal equations have a closed form (this is Least Squares Regression), so the fit is exact in one solve; NumPy polyfit is the direct tool there.
TF7. Passing c unchanged to linprog maximizes profit.
False —
linprog only minimizes, so maximizing requires passing and then reporting -res.fun.TF8. res.success == True guarantees the answer is the one you wanted.
False —
success only means the solver converged, not that it converged to the correct or global point; a bad x0 can converge happily to the wrong root or a saddle.TF9. A stationary point () is always a minimum.
False — it may be a maximum or a saddle; you need the Hessian to be positive semi-definite there to confirm a minimum.
Spot the error
SE1. sol = fsolve(F, x0=[1,1]) where F returns [x**2 + y**2, x - y] to solve .
Error — the constants were dropped;
fsolve drives the vector to zero, so you must move everything to one side: [x**2 + y**2 - 25, x - y - 1].SE2. To maximize : res = linprog([40,30], A_ub=..., b_ub=...).
Error — this minimizes , giving the worst plan (likely the origin); negate to
c=[-40,-30] and read profit as -res.fun.SE3. Fitting with curve_fit(model, x, y) and no p0.
Error — a nonlinear model needs a starting guess; the default
p0=[1,1,...] can land in a flat region and return garbage or raise a runtime error. Supply a physical p0.SE4. Zeroing a system of equations by calling minimize on the sum of the equation expressions.
Error —
minimize drives the sum to its smallest value, which can be very negative rather than zero, so individual equations need not be satisfied; use fsolve (or minimize the sum of squares of the residuals).SE5. Reading the minimum value as print(res) and expecting a number.
Error —
res is an OptimizeResult object; the location is res.x, the value is res.fun, and convergence is res.success.SE6. Encoding by adding a row [-1,0] to A_ub with b_ub entry 0, then also leaving bounds at its default.
Not wrong per se, but redundant —
bounds=[(0,None),...] already enforces ; duplicating it as an inequality just clutters the model. Prefer bounds.SE7. Trusting pcov from curve_fit after the fit clearly failed to converge.
Error — when the fit is unreliable
pcov can contain inf or meaningless values; check convergence and residuals before quoting np.sqrt(np.diag(pcov)) as uncertainties.Why questions
WHY1. Why do minimize and fsolve both need a starting guess but linprog does not?
The first two follow the local slope/curvature and can only see nearby structure, so where you start decides where you land; a linear program's landscape has a constant gradient with the optimum forced to a corner, so the solver enumerates vertices systematically instead.
WHY2. Why does Newton's method appear inside all four engines?
Each engine ultimately linearizes a function around the current point and solves that linear model for the next step; that linearize-and-solve loop is Newton's method (see Newton's Method), with the derivative replaced by a Jacobian or Hessian.
WHY3. Why is a linear objective's optimum always at a corner?
A linear function has constant gradient, so it never flattens inside the feasible region — you can always keep sliding downhill along a wall until walls block you on all sides, and that intersection of walls is a vertex.
WHY4. Why do we square residuals instead of taking absolute values?
Squares are smooth and differentiable everywhere (so gradient methods work) and correspond to Gaussian maximum likelihood; absolute values are non-smooth at zero, which is why least squares is the default rather than least-absolute-deviation.
WHY5. Why can two different x0 values give two different fsolve answers for the same system?
The system has multiple roots, and Newton-type iteration converges to whichever root's basin of attraction contains the seed, so
[1,1] and [-5,-5] can land on and respectively.WHY6. Why is convexity such a prized property in optimization?
For a convex objective every local minimum is the global minimum, so a local solver's answer is trustworthy without restarts — this is the whole promise of Convex Optimization.
WHY7. Why does curve_fit return pcov alongside the parameters?
The diagonal of the covariance matrix gives the variances of the fitted parameters, so
np.sqrt of it yields free error bars — a fit without uncertainty is only half an answer.Edge cases
EC1. What happens if you give minimize a starting point exactly on a saddle where ?
The gradient is already zero, so a pure gradient method may report "converged" and never leave the saddle; the Hessian would reveal it is not a minimum, and a small perturbation of
x0 escapes it.EC2. What does fsolve do if the Jacobian is singular at the guess (derivative effectively zero)?
The Newton step blows up or fails because you cannot invert a singular Jacobian; the solver may stall or diverge, signalled by
ier != 1 from full_output=True.EC3. What if a linear program's feasible region is unbounded in the descent direction?
There is no finite optimum, so
linprog reports status "problem is unbounded" — the objective can be driven to and no vertex is optimal.EC4. What if the linear constraints are contradictory (empty feasible region)?
linprog returns an infeasibility status because no satisfies all constraints at once; there is nothing to optimize.EC5. What if you curve_fit with fewer data points than parameters?
The problem is underdetermined — infinitely many parameter sets fit exactly, so the solver either fails or returns an arbitrary one, and
pcov cannot give meaningful uncertainties.EC6. What if the objective handed to minimize is flat (constant) everywhere?
The gradient is zero everywhere, so the very first point already "satisfies" the stopping test;
minimize returns x0 unchanged because no step improves anything.EC7. What if curve_fit's p0 sits at a point where the model is numerically flat in a parameter (e.g. huge so )?
That parameter's gradient vanishes, so the solver gets no signal to move it and may leave it stuck at the bad guess — a reason to seed
p0 in a sensible, active range.EC8. What happens if a root you seek lies exactly at a tangency where touches zero without crossing?
The function value and its slope are both near zero there, so Newton-type steps shrink very slowly (linear instead of quadratic convergence) and may report a near-root with larger residual than expected.