scipy.optimize — minimize, fsolve, curve_fit, linprog: Visual walkthrough

We follow the parent's own Newton's Method box, but slowly, with a picture per step.

Step 0 — What is a "zero", and why do we chase it?

Before any formula, let's agree on the goal in plain words.

The picture below shows one curve crossing the flat line. That crossing point is what all four tools ultimately search for — directly (fsolve) or through the back door (minimize chases the zero of the slope).

Figure — scipy.optimize — minimize, fsolve, curve_fit, linprog

WHY this matters: you can rarely solve $g (x) = 0$ with algebra when $g$ is messy. So instead of solving, we will sneak up on the crossing point step by step.

Step 1 — The slope: the only local information we have

Standing at some guess $x_{k}$ (the little $k$ just counts our guesses: guess number 0, 1, 2, …), we can measure two things about the curve right where we stand:

its height $g (x_{k})$ — how far above or below the line we are, and
its slope $g^{'} (x_{k})$ — how steeply the curve is tilting there.

Definition Slope

$g^{'} (x)$ — a limit of "rise over run" Pick your point $x_{k}$ and a second point a tiny distance $h$ to its right, at $x_{k} + h$ . Draw the straight line through those two points on the curve; its steepness is the plain schoolyard ratio $\frac{rise}{run} = \frac{g ( x _{k} + h ) - g ( x _{k} )}{h} .$ This is a difference quotient — literally "how much height changed, divided by how far right we stepped." Now shrink $h$ toward $0$ : the two points slide together and that chord snuggles onto the curve, becoming the tangent line. The number it settles on is the slope $g^{'} (x_{k})$ (read "g-prime at $x_{k}$ "): $g^{'} (x_{k}) = lim_{h \to 0} \frac{g ( x _{k} + h ) - g ( x _{k} )}{h} .$ The symbol $lim_{h \to 0}$ just means "the value this ratio homes in on as $h$ gets vanishingly small." Steep uphill → big positive number; steep downhill → big negative; flat → zero.

The figure shows the curve (cyan) and, in amber, the straight tangent ramp at $x_{k}$ .

WHY a tangent line? Because a straight line is the only thing we can solve $= 0$ instantly. The curve is hard; a ramp is easy. So we replace the curve by its ramp near $x_{k}$ and solve the easy problem instead.

Step 2 — Replace the curve with its ramp (the Taylor idea)

Here is the one leap of faith, stated with zero jargon: close to where you stand, a smooth curve and its tangent ramp are almost the same line. Zoom in enough and every curve looks straight.

Writing "a small sideways step" as $Δ$ (Greek letter delta, meaning "a little change"), the ramp's height after stepping is:

This is the parent note's "keep linear terms of the Taylor expansion" — but now you can see it: we threw away the curve's bending and kept only its straight part.

Step 3 — Demand the ramp's height $= 0$ and solve for the step

We want to land on the horizontal line. We cannot force the real curve to $0$ in one clean move — but we can force the ramp (our straight stand-in) to $0$ , because a straight line meets the axis at exactly one solvable point. So we set the approximate height to zero:

Read the fraction as pure geometry: "how far to walk = (height error) ÷ (tilt)." A tall error over a gentle tilt means a long walk; a small error over a steep tilt means a tiny nudge. That is the ramp sliding down to the axis. Because we zeroed the ramp and not the curve, $x_{k} + Δ$ lands close to the true zero, not exactly on it — which is why we must repeat.

Step 4 — The Newton update: take the step, repeat

Our new, better guess is the old spot plus the step $Δ$ :

Now the trick: the ramp only approximated the curve, so $x_{k + 1}$ isn't the exact zero — but it's closer (once we're near enough). So we stand there, draw a fresh tangent, slide again, and again. Each pass squeezes the gap.

Intuition WHY it converges so fast (the "squaring" of the error, seen)

The only reason $x_{k + 1}$ misses the true zero is the curve's bending — the part we discarded in Step 2. Near the zero the leftover error behaves like the area of a tiny square whose side is the current error: if you are a distance $e$ away, the ramp's mismatch grows like $e \times e = e^{2}$ (bending $\times$ how far you walked across it). So an error of $0.1$ becomes about $0.01$ , then $0.01$ becomes about $0.0001$ — each step roughly squares the previous error. That "error → error $^{2}$ " behaviour is what people call quadratic convergence, and it is why Hessian-based [[Gradient and Hessian|minimize]] reaches machine precision in a handful of iterations. (Two warnings: this only kicks in once you are already close — see Step 8 — and it also needs the tilt at the true zero to be non-zero — see Step 9.)

Step 5 — From root-finding to minimizing (the parent's big reveal)

The parent note's headline: minimizing $f$ = finding a root of its slope $f^{'}$ . Watch why, as a picture.

So to minimize $f$ , just run Newton's method on $g = f^{'}$ . Substitute $g = f^{'}$ (so $g^{'} = f^{''}$ , the "slope of the slope", i.e. curvature):

Common mistake Zero slope is not always a

minimum — check the sign of $f^{''}$ Newton on $f^{'}$ chases any flat spot: a valley bottom (minimum), a hilltop (maximum), or a level ledge (saddle/inflection) all have $f^{'} = 0$ . The formula cannot tell them apart on its own. The tell is the curvature:

$f^{''} (x) > 0$ → curve bends upward (a bowl) → genuine minimum. ✅
$f^{''} (x) < 0$ → curve bends downward (a dome) → you have converged to a maximum. ❌
$f^{''} (x) = 0$ → flat curvature → possible inflection/saddle, and the step also divides by zero. ❌

Fix: real solvers guard against this — e.g. Newton-CG and trust-region methods force the effective curvature to stay positive so the step always points downhill. In multiple dimensions the same check becomes "the Hessian must be positive-definite," the exact stationary-point condition the parent note states.

Step 6 — Edge case A: the slope is zero (flat spot, division blows up)

Every honest walkthrough must cover the failures. Look at the update: we divide by $g^{'} (x_{k})$ . What if the tilt is $0$ ?

Step 7 — Edge case B: wrong starting guess → the other root

The method is local: it slides to whatever zero its ramps happen to point at. Different seeds can land on different roots — exactly the parent's fsolve warning ( $[1, 1] \to (4, 3)$ but $[- 5, - 5] \to (- 3, - 4)$ ).

WHY linprog doesn't have this problem: a linear objective has a constant slope that never flattens, so there's no local trap — the optimum is always a corner, found globally. That's the whole reason it's a separate tool (see Linear Programming Simplex and Convex Optimization).

Step 8 — Edge case C: a bad guess makes Newton diverge

The "error squares" magic of Step 4 was promised only when you start close. Far away, the very same tangent-slide can throw you further from the zero — or into an endless loop.

Common mistake Newton can run away entirely

Two classic traps, both visible as a mispointed tangent ramp:

Near an inflection / gentle slope: if the curve is nearly flat where you stand (tiny $g^{'} (x_{k})$ ), the ramp is almost horizontal and $Δ = - g / g^{'}$ becomes huge — one step hurls you far across the axis, often into a worse region than you started.
Overshoot / cycling: with strong curvature, the ramp can point past the zero to a spot whose tangent points right back, so the iterates bounce $x_{0} \to x_{1} \to x_{0} \to \dots$ forever without settling.

Why it feels wrong: we derived that each step improves — but that proof quietly assumed the ramp was a good stand-in, i.e. that you were already near the zero. Break that assumption and the guarantee evaporates. Fix: this is precisely why SciPy wraps Newton in a line search (shorten the step until it actually reduces $∣ g ∣$ ) or a trust region (never trust the ramp beyond a small radius). Practically: pick a sensible x0, and if it diverges, restart closer.

Step 9 — Edge case D: a multiple root ( $g^{'} (x^{*}) = 0$ ) — convergence slows to a crawl

Steps 6–8 were about the tilt vanishing where we stand. Now the subtler trap: the tilt vanishing at the very zero we're chasing. This happens when the curve doesn't cleanly cross the axis but merely touches it — a double (multiple) root, like $g (x) = (x - 2)^{2}$ at $x^{*} = 2$ .

Common mistake A "kissed" zero destroys the fast

$e \to e^{2}$ speed-up At a simple root the curve slices through the axis with a healthy non-zero tilt $g^{'} (x^{*}) \neq = 0$ — that non-zero tilt is exactly what the $e \to e^{2}$ argument of Step 4 leaned on. At a multiple root the curve flattens as it reaches the zero: $g^{'} (x^{*}) = 0$ . Now the Newton step $- g / g^{'}$ is a tiny-over-tiny ratio ( $\frac{0}{0}$ in the limit), and the error no longer squares. Instead it merely gets multiplied by a constant fraction each step: $e_{k + 1} \approx (1 - \frac{1}{m}) e_{k}$ where $m$ is the multiplicity (how many times the root is repeated: $m = 2$ for a touch, $m = 3$ for a flatter kiss…). For a double root that factor is $1 - \frac{1}{2} = \frac{1}{2}$ : the error only halves each step instead of squaring — this is linear convergence, dramatically slower, and rounding noise near the flat spot can stall or wobble the last digits entirely. Fix: if you know the multiplicity, the modified Newton step $x_{k + 1} = x_{k} - m g / g^{'}$ restores quadratic speed; otherwise fsolve may report success with a loose tolerance, so tighten xtol and sanity-check the residual $g (x)$ .

Step 10 — What `fsolve` actually runs (not plain Newton)

We derived Newton because it is the clearest lens. But it would be dishonest to claim fsolve is Newton-with-a-leash.

Definition

fsolve = Powell's hybrid method (MINPACK hybrd/hybrj) scipy.optimize.fsolve wraps the Fortran routine MINPACK hybrd. It blends two moves each iteration:

a Newton/Gauss–Newton step (the ramp-slide we built) when the local model looks trustworthy, and
a cautious gradient-descent-like step inside a trust region (a "dogleg") when it doesn't — this is the Powell hybrid idea: interpolate between the bold Newton jump and a timid safe step depending on how much the last step actually helped.

Two more practical facts the derivation hid:

You usually don't supply a derivative. fsolve estimates the Jacobian $J$ (the multidimensional $g^{'}$ ) by finite differences — nudging each input by a tiny $h$ and measuring the output change, exactly the difference quotient from Step 1. Supplying an exact fprime is faster and steadier.
It reuses/updates that Jacobian estimate across steps rather than recomputing from scratch (a rank-1 Broyden-style update), which is why it stays cheap in high dimensions.

So: the ramp-slide is the beating heart, but fsolve sheathes it in a trust-region hybrid with a finite-difference Jacobian — that is what makes it robust to the far-away and flat-spot failures of Steps 6, 8, and 9.

The one-picture summary

Everything above, compressed: a curve, one guess, one tangent ramp, one slide to the axis, repeat — and the same picture reused with $g = f^{'}$ to find a valley bottom.

Recall Feynman: the whole walkthrough in plain words

You want to find where a wiggly line crosses the floor, but the line's too complicated to solve. So you cheat: stand at a guess, feel how steeply the line tilts (that tilt is the slope — imagine two dots on the curve sliding together until the chord between them becomes a perfect kiss), and pretend the curve is a straight ramp from there. A straight ramp is easy — you can instantly see where it hits the floor, so you jump to that spot. The ramp lied a little because the real line was curved, so you're not there yet, but you're closer — and because the only leftover is that little bending, the miss shrinks like a square each time, so a few jumps and you're basically dead on. That's the engine inside fsolve. To find the bottom of a valley instead, notice the bottom is where the ground is flat — where the slope hits zero — so play the exact same game on the slope, but check the ground actually curves upward there (a bowl, not a dome or a ledge) or you'll happily land on a peak. That's minimize. Four warnings: if you stand on a flat spot your ramp is horizontal and points nowhere (divide-by-zero); if the line crosses twice, where you start decides which crossing you fall into; if you start too far off, the ramp can fling you into the wilderness; and if the line merely kisses the floor instead of slicing through it, the fast squaring dies and progress crawls to a halt. Real tools don't run raw Newton — fsolve uses Powell's hybrid scheme, mixing bold Newton jumps with cautious safe steps and guessing the slopes by tiny finite-difference nudges. linprog skips all of it, because straight walls always send you to a corner.

Recall Quick check

Newton update in one line ::: $x_{k + 1} = x_{k} - g (x_{k}) / g^{'} (x_{k})$ — slide down the tangent to the axis. The slope $g^{'} (x_{k})$ is defined as… ::: the limit of the difference quotient $\frac{g ( x _{k} + h ) - g ( x _{k} )}{h}$ as $h \to 0$ — a chord tightening into the tangent. Why does the same formula minimize? ::: Put $g = f^{'}$ ; the valley bottom is where the slope $f^{'} = 0$ , a root of $f^{'}$ . What extra check turns a stationary point into a real minimum? ::: The curvature must be positive, $f^{''} (x) > 0$ (a bowl, not a dome or ledge). What breaks Newton at a flat point? ::: $g^{'} (x_{k}) = 0$ ⇒ horizontal ramp ⇒ divide-by-zero ⇒ step to infinity. Why can two seeds give different roots? ::: Newton is local; its tangents chain to the nearest crossing, and a curve may cross more than once. Why can a far-off guess diverge? ::: The "error squares" guarantee needs you to start close; far away the ramp is a poor stand-in and can fling iterates away or cycle. What slows Newton to linear convergence? ::: A multiple root where $g^{'} (x^{*}) = 0$ (the curve only touches the axis): the error halves instead of squaring. What does fsolve actually implement? ::: Powell's hybrid method (MINPACK hybrd) — Newton mixed with a trust-region safe step, using a finite-difference Jacobian.

Flashcards

Derive the Newton step from a tangent line.

Approximate

g (x_{k} + Δ) \approx g (x_{k}) + g^{'} (x_{k}) Δ

, set that approximation to

0

, solve:

Δ = - g (x_{k}) / g^{'} (x_{k})

Define the slope

g^{'}

that Newton needs.

The limit

lim_{h \to 0} \frac{g ( x _{k} + h ) - g ( x _{k} )}{h}

— the difference quotient (rise/run) as the run shrinks to zero.

Geometric meaning of one Newton step.

Slide down the tangent ramp at

x_{k}

until it hits the horizontal axis; that landing

x

x_{k + 1}

Why is minimizing

f

a root problem, and what sign check is needed?

The minimum is where slope

f^{'} = 0

, so run Newton on

g = f^{'}

x_{k + 1} = x_{k} - f^{'} / f^{''}

— and require

f^{''} > 0

so it's a minimum, not a maximum or saddle.

When does Newton lose its quadratic speed?

At a multiple root

g^{'} (x^{*}) = 0

— the curve only touches the axis, so error shrinks linearly (halves per step for a double root) instead of squaring.

Does fsolve run plain safeguarded Newton?

No — it wraps MINPACK's Powell hybrid method, blending Newton steps with trust-region safe steps and a finite-difference Jacobian.