Intuition The one core ideaEvery problem in scipy.optimize — best, root, fit, allocate — is really the same act: stand on a landscape of numbers and keep stepping until you can't improve. To read that sentence properly you only need to understand a height map (a function), which direction is downhill (the gradient), and what "can't improve" means (a flat spot, the zero of a slope).
This page assumes nothing . If the parent note wrote a symbol, we build it here from a picture first. Read top to bottom — each idea is the floor the next one stands on.
Before any fancy symbol, fix the mental image.
f function f position ), it returns ONE number (a height ). Write f ( x ) x
Look at the figure. The horizontal axis is where you stand (x height above each point is the value f ( x ) lowest point — the minimum . Everything in this whole topic is a hunt for special points on such a map.
minimize looks for the lowest point of this landscape. fsolve looks for where a different curve crosses the flat sea-level line y = 0
The parent writes f ( x ) bold x
x x line . A ==bold x list of numbers, e.g. x = ( x 1 , x 2 ) plane (or in higher-dimensional space). The bold font is a warning: "this may be more than one number".
In the figure the arrow points from the origin to the position x = ( x 1 , x 2 ) above the whole floor , like a real hill over a field. The parent's bowl f ( x , y ) = ( x − 3 ) 2 + ( y + 1 ) 2
To go downhill you must know which way tilts down . In 1D that tilt is the derivative.
f ′ ( x ) derivative f ′ ( x ) how fast the height changes if you nudge x zero = flat .
Intuition Why "zero = flat" is the whole gameAt the very bottom of a valley the ground is momentarily level — you'd measure zero tilt. So the lowest point satisfies f ′ ( x ) = 0 f ′ = 0 "minimizing f
The notation f ′ d x df f x
On a field (2D) there isn't one slope — the ground tilts differently as you face different ways. We need a slope per coordinate .
∇ f gradient ∇ f vector whose entries are the slope in each direction:
∇ f = ( ∂ x 1 ∂ f , ∂ x 2 ∂ f , … ) in the direction the hill rises fastest . Its opposite, − ∇ f straight downhill .
The ∂ f ′ one coordinate at a time , holding the others frozen. Look at the figure: red arrows are the gradient at several points, always pointing away from the low centre. At the very bottom the arrow shrinks to nothing — ∇ f = 0
Common mistake "A flat spot is always a minimum."Why it feels right: the bottom of a bowl is flat, so flat must mean bottom. Truth: the top of a hill and a saddle are also flat (∇ f = 0
Flat isn't enough. Is the flat spot a valley (curves up ) or a peak (curves down )?
Definition Second derivativef ′′ f ′′ ( x ) of the slope . If f ′′ > 0 cup-shaped (bottom of a valley); if f ′′ < 0 cap-shaped (top of a hill).
H Hessian H Positive semi-definite " is the multi-D way of saying "curves upward no matter which way you look" — the test for a genuine minimum.
You don't need to compute a Hessian by hand here; you need the picture : gradient tells you which way to step, curvature tells you how far and whether you've found a true bottom. Newton's method (parent's engine) uses both.
fsolve doesn't minimize — it hunts for crossings .
Definition Root — where a curve hits zeroA root of g 0 g ( x ) = 0 crossing the horizontal sea-level line .
Intuition Why everything is moved to one sideThe parent writes x 2 + y 2 = 25 x 2 + y 2 − 25 fsolve only knows how to find where its function returns zero . So any equation "left = right" becomes "left − right = 0". Bold F several such equations stacked into a list, one entry per equation.
F ( x ) = 0 means every entry of the list is zero at the same time.
When F list of functions, its slope is a grid of slopes.
J Jacobian J F f J − 1 J 1/ g ′
You only need to recognise the word: when the parent writes Δ = − J − 1 F "one Newton step for a system" — no new idea, just the plural of a slope.
curve_fit bends a model curve toward scattered dots.
r i ( x i , y i ) y ^ i residual r i = y i − y ^ i
The red segments in the figure are the residuals. The subscript i i ∑ i = 1 N add up over all N :
∑ i = 1 N r i 2 = r 1 2 + r 2 2 + ⋯ + r N 2
Intuition Why square each gapIf we added the raw gaps r i cancel , hiding a bad fit. Squaring makes every miss positive, so the total honestly measures total wrongness . Minimizing that total is the least-squares idea → see Least Squares Regression .
The parameters we tune are bundled as θ a , b a e − b x
linprog's symbols are the tamest — everything is a straight line or flat plane.
c A b
c ⊤ x weighted sum of your decisions: cost/profit per unit times how many units. The little ⊤ A x ≤ b limits (budget, capacity), each cutting off a region.
Intuition Why the answer sits at a cornerA linear objective tilts the whole floor like a perfectly flat ramp — it never flattens out inside the allowed region. So you keep sliding downhill until a wall stops you, then slide along the wall until a second wall stops you: a corner . That's why linprog only checks corners (see Linear Programming Simplex and Convex Optimization ).
Stationary point del f = 0
Curvature f'' and Hessian
Residuals and sum of squares
Read it downward: the plain height-map idea splits into slope and position ; those merge into the gradient ; the gradient being zero is what every engine hunts for.
Cover the right side and test yourself. If any answer is fuzzy, reread that section before the parent note.
What does f ( x ) A single height above the point x
Why is x It is a list of coordinates — a position in possibly many dimensions, not one number.
What does the derivative f ′ ( x ) The steepness/tilt of the curve — how fast height changes as you nudge x
What does f ′ = 0 The ground is momentarily flat — a candidate lowest (or highest) point.
What is the gradient ∇ f An arrow on the floor pointing in the steepest-uphill direction; downhill is − ∇ f
Why is minimizing f ∇ f Because a minimum is a flat spot, and flat means ∇ f = 0
What extra info does curvature (f ′′ Whether a flat spot curves up (minimum), down (maximum), or is a saddle .
Why do we rewrite an equation as "something = 0 fsolve? fsolve only finds where its function returns zero , so we move all terms to one side.
What is a residual r i The vertical gap y i − y ^ i
Why square the residuals before summing? So positive and negative misses don't cancel , giving an honest total error.
Why does a linear program's optimum sit at a corner? A linear objective never flattens inside the region, so you slide along walls until a vertex stops you.
