scipy.optimize — minimize, fsolve, curve_fit, linprog: Exercises

The four engines in one glance

Before we start, a picture of what problem each tool solves. Look at each panel: the little dot is the answer the tool hunts for.

Figure — scipy.optimize — minimize, fsolve, curve_fit, linprog

Level 1 — Recognition

L1·Q1 — Which tool?

For each task, name the single best scipy.optimize function.

(a) Find the smallest value of $f (x) = x^{4} - 3 x^{2} + 2$ . (b) Find every $x$ where $cos x = x$ . (c) Choose $a, b$ so $y = a sin (b x)$ hugs 200 noisy sensor readings. (d) Maximize $5 x_{1} + 4 x_{2}$ subject to $6 x_{1} + 4 x_{2} \leq 24$ , $x \geq 0$ .

Recall Solution

(a) minimize — we want the smallest value of a scalar objective. (b) fsolve — rewrite as $cos x - x = 0$ ; we want a root. (c) curve_fit — we are fitting a model to data (least squares). (d) linprog — linear objective, linear constraints ⇒ the answer lives at a corner.

The tell-tale words: smallest → minimize, equals zero / solve → fsolve, fit data → curve_fit, linear + constraints → linprog.

L1·Q2 — Read the result object

After res = minimize(f, x0), which attribute holds the winning input, which holds the winning score, and which is the True/False flag that it worked?

Recall Solution

res.x ::: the solution vector $x^{*}$ (the winning input).
res.fun ::: the objective value $f (x^{*})$ (the winning score).
res.success ::: boolean flag — True if the solver believes it converged.

Level 2 — Application

L2·Q1 — Minimize a 1D quartic

Minimize $f (x) = x^{4} - 3 x^{2} + 2$ starting from x0=2. What is $x^{*}$ and $f (x^{*})$ ?

Recall Solution

WHAT: we look for a stationary point where the slope $f^{'} (x) = 0$ . $f^{'} (x) = 4 x^{3} - 6 x = 2 x (2 x^{2} - 3) = 0 \Rightarrow x = 0 or x = \pm 3/2 .$ WHY those: setting the derivative to zero finds flat spots — candidates for minima. $x = 0$ is a local max (a hilltop), and $x = \pm 3/2 \approx \pm 1.2247$ are the two valleys. Starting at x0=2 (right side), the iteration rolls down into the right valley $x^{*} = 3/2 \approx 1.2247$ . $f (3/2) = (3/2)^{2} - 3 (3/2) + 2 = 2.25 - 4.5 + 2 = - 0.25.$

from scipy.optimize import minimize
f = lambda x: x[0]**4 - 3*x[0]**2 + 2
res = minimize(f, x0=[2.0])
print(res.x)    # ≈ [1.2247]
print(res.fun)  # ≈ -0.25

Answer: $x^{*} \approx 1.2247$ , $f (x^{*}) = - 0.25$ .

L2·Q2 — fsolve a nonlinear pair

Solve $x^{2} + y^{2} = 25$ and $x - y = 1$ from x0=[1,1].

Recall Solution

WHY move to one side: fsolve seeks where the returned vector is $0$ , so rewrite each equation as something = 0.

from scipy.optimize import fsolve
def F(v):
    x, y = v
    return [x**2 + y**2 - 25, x - y - 1]
print(fsolve(F, x0=[1,1]))   # ≈ [4., 3.]

By hand: $x = y + 1$ , so $(y + 1)^{2} + y^{2} = 25 \Rightarrow 2 y^{2} + 2 y - 24 = 0 \Rightarrow y^{2} + y - 12 = 0 \Rightarrow (y + 4) (y - 3) = 0$ . Roots $y = 3$ (giving $x = 4$ ) or $y = - 4$ (giving $x = - 3$ ). The seed [1,1] is nearer the first root. Answer: $(x, y) = (4, 3)$ .

L2·Q3 — linprog a tiny plan

Maximize $5 x_{1} + 4 x_{2}$ with $6 x_{1} + 4 x_{2} \leq 24$ , $x_{1} + 2 x_{2} \leq 6$ , $x \geq 0$ .

Recall Solution

WHY negate: linprog only minimizes, so to maximize $c^{⊤} x$ we minimize $- c^{⊤} x$ .

from scipy.optimize import linprog
c = [-5, -4]
A_ub = [[6,4],[1,2]]
b_ub = [24, 6]
res = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=[(0,None),(0,None)])
print(res.x)      # ≈ [3., 1.5]
print(-res.fun)   # 21.0

Corner-hunting by hand: the two active constraints intersect where $6 x_{1} + 4 x_{2} = 24$ and $x_{1} + 2 x_{2} = 6$ . Multiply the second by 2: $2 x_{1} + 4 x_{2} = 12$ . Subtract from the first: $4 x_{1} = 12 \Rightarrow x_{1} = 3$ , then $x_{2} = (6 - 3) /2 = 1.5$ . Objective $= 5 (3) + 4 (1.5) = 21$ . Answer: $x = (3, 1.5)$ , profit $= 21$ .

Level 3 — Analysis

L3·Q1 — One Newton step by hand

fsolve is built on Newton's method: $x_{k + 1} = x_{k} - g (x_{k}) / g^{'} (x_{k})$ . For $g (x) = x^{2} - 2$ starting at $x_{0} = 1$ , compute $x_{1}$ and $x_{2}$ by hand. What are they converging to?

Recall Solution

WHAT Newton does: it replaces the curve by its tangent line at $x_{k}$ and jumps to where that line crosses zero — a picture worth holding in mind. $g^{'} (x) = 2 x$ . $x_{1} = 1 - \frac{1 ^{2} - 2}{2 \cdot 1} = 1 - \frac{- 1}{2} = 1.5.$ $x_{2} = 1.5 - \frac{1. 5 ^{2} - 2}{2 \cdot 1.5} = 1.5 - \frac{0.25}{3} = 1.5 - 0.08333 \dots = 1.41 \overline{6} .$ It is racing toward $2 \approx 1.41421$ . Notice how fast: $x_{2}$ already matches $2$ to three decimals. That doubling of correct digits per step is Newton's signature. See Newton's Method. Answer: $x_{1} = 1.5$ , $x_{2} = 1.41 \overline{6}$ , converging to $2$ .

L3·Q2 — Why the seed decides the root

For the system in L2·Q2, predict which root fsolve returns from x0=[-5,-5], and explain geometrically.

Recall Solution

WHY the seed matters: Newton-type solvers are local — they follow the tangent downhill from the seed and lock onto the nearest root, never scanning globally. The circle $x^{2} + y^{2} = 25$ meets the line $x - y = 1$ at two points: $(4, 3)$ (upper-right) and $(- 3, - 4)$ (lower-left). The seed $(- 5, - 5)$ sits in the lower-left, closest to $(- 3, - 4)$ . Answer: fsolve returns $(- 3, - 4)$ . Look at the figure: each seed's basin of attraction is the side of the line it starts on.

L3·Q3 — Least-squares line by the normal equations

Fit $y = m x + c$ to the three points $(0, 1), (1, 3), (2, 3)$ . Minimize $S = \sum (y_{i} - m x_{i} - c)^{2}$ analytically. Give $m, c$ .

Recall Solution

WHY set derivatives to zero: $S$ is a smooth bowl in $(m, c)$ ; its lowest point has both partial slopes zero. $\frac{\partial S}{\partial c} = 0$ and $\frac{\partial S}{\partial m} = 0$ give the normal equations: $N c + m \sum x_{i} = \sum y_{i}, c \sum x_{i} + m \sum x_{i}^{2} = \sum x_{i} y_{i} .$ Numbers: $N = 3$ , $\sum x = 3$ , $\sum x^{2} = 5$ , $\sum y = 7$ , $\sum x y = 0 \cdot 1 + 1 \cdot 3 + 2 \cdot 3 = 9$ . $3 c + 3 m = 7, 3 c + 5 m = 9.$ Subtract: $2 m = 2 \Rightarrow m = 1$ ; then $3 c = 7 - 3 = 4 \Rightarrow c = 4/3$ . This is exactly what Least Squares Regression and NumPy polyfit compute in closed form — no iteration needed because the model is linear. Answer: $m = 1$ , $c = 4/3 \approx 1.3333$ .

Level 4 — Synthesis

L4·Q1 — Same problem, two tools

Find the minimum of $f (x) = (x - 4)^{2} + 1$ twice: once with minimize, once by handing $f^{'} (x) = 0$ to fsolve. Show both give the same $x$ .

Recall Solution

The unifying idea: minimizing $f$ = finding a root of $f^{'}$ . Two APIs, one skeleton. $f^{'} (x) = 2 (x - 4) = 2 x - 8$ . Root of $f^{'}$ is $x = 4$ ; that is also the valley bottom, where $f = 1$ .

from scipy.optimize import minimize, fsolve
f  = lambda x: (x[0]-4)**2 + 1
df = lambda x: [2*(x[0]-4)]
print(minimize(f, x0=[0]).x)   # ≈ [4.]
print(fsolve(df, x0=[0]))      # ≈ [4.]

Answer: both return $x^{*} = 4$ ; $f (4) = 1$ . The Hessian $f^{''} = 2 > 0$ confirms it is a minimum (see Gradient and Hessian).

L4·Q2 — Constrained by hand, checked by linprog

A bakery makes cakes ( $x_{1}$ ) and pies ( $x_{2}$ ), profit $\$ 3 $an d$ $2 $. O v e n t im e :$ x_1+x_2\le 100 $. F l o u r :$ 2x_1+x_2\le 150 $. A l l$ \ge 0$. Find the best mix and profit.

Recall Solution

WHY corners: a linear objective never flattens inside the region — its gradient is constant — so you slide along the walls to a vertex (see Linear Programming Simplex, Convex Optimization). Candidate corners of the feasible polygon:

$(0, 0)$ : profit $0$ .
$(0, 100)$ (oven wall, $x_{1} = 0$ ): profit $200$ .
$(75, 0)$ (flour wall, $x_{2} = 0$ ): profit $225$ .
Intersection of oven & flour: $x_{1} + x_{2} = 100$ , $2 x_{1} + x_{2} = 150$ . Subtract ⇒ $x_{1} = 50$ , $x_{2} = 50$ . Profit $3 (50) + 2 (50) = 250$ . The intersection corner wins.

from scipy.optimize import linprog
res = linprog([-3,-2], A_ub=[[1,1],[2,1]], b_ub=[100,150],
              bounds=[(0,None),(0,None)])
print(res.x)     # ≈ [50., 50.]
print(-res.fun)  # 250.0

Answer: 50 cakes, 50 pies, profit $\$ 250$.

Level 5 — Mastery

L5·Q1 — Exponential fit, error bars

You fit $y = a e^{- b x}$ with curve_fit and get pcov = [[0.0016, 0], [0, 0.0009]]. Report the $1 σ$ uncertainty on $a$ and $b$ . What does a large off-diagonal entry mean?

Recall Solution

WHY the square root: pcov is the covariance matrix; its diagonal holds variances $σ^{2}$ . The uncertainty (standard deviation) is the square root. $σ_{a} = 0.0016 = 0.04, σ_{b} = 0.0009 = 0.03.$

import numpy as np
pcov = [[0.0016,0],[0,0.0009]]
print(np.sqrt(np.diag(pcov)))   # [0.04 0.03]

A large off-diagonal entry means $a$ and $b$ are correlated — the fit can't pin them down independently (raising $a$ while lowering $b$ gives a similar curve). Here the off-diagonals are $0$ , so they are independent. Answer: $σ_{a} = 0.04$ , $σ_{b} = 0.03$ .

L5·Q2 — Where Newton stalls

Explain what happens when fsolve (Newton at heart) tries $g (x) = x^{2}$ from $x_{0} = 3$ . Compute two steps and describe the failure/slowdown.

Recall Solution

The subtlety: $x = 0$ is a double root ( $g$ and $g^{'}$ both vanish there). Newton's fast doubling of digits assumes a simple root; at a double root it crawls, roughly halving the error each step (linear, not quadratic). $g^{'} (x) = 2 x$ . Update $x_{k + 1} = x_{k} - x_{k}^{2} / (2 x_{k}) = x_{k} - x_{k} /2 = x_{k} /2$ . $x_{1} = 3/2 = 1.5, x_{2} = 1.5/2 = 0.75.$ It keeps halving toward $0$ — it does converge, but slowly, and fsolve may report near-success with a residual that shrinks lazily. Answer: $x_{1} = 1.5$ , $x_{2} = 0.75$ ; convergence is merely linear (halving) because of the double root.

L5·Q3 — Diagnose a bad fit

Someone runs curve_fit(model, x, y) on a nonlinear model with no p0, and gets nonsense (or a RuntimeError). State the cause and the one-line fix.

Recall Solution

Cause: with no p0, curve_fit defaults every parameter to $1$ . Nonlinear least squares is non-convex — the error surface has many valleys — so a poor seed can slide into a wrong valley or fail to converge. Contrast: linear fits like NumPy polyfit need no seed because their error surface is a single bowl (convex), solvable in closed form. Fix: supply a physically sensible seed, e.g. curve_fit(model, x, y, p0=[y.max(), 1.0]). Answer: default seed $[1, 1, \dots]$ + non-convex surface → divergence; fix with a good p0.

Recall Feynman recap: the whole ladder in one breath

You learned to spot the right engine (L1), drive each one (L2), see why Newton and least-squares work the way they do (L3), bridge two tools on one problem (L4), and diagnose the quiet failure modes — double roots, missing seeds, deceptive residuals (L5). One skeleton underneath it all: guess, look at the local shape, step toward better, repeat.