Intuition The one big ideaAlmost every "find the best / find the root / fit the data / allocate resources" problem reduces to searching a number space until a condition is satisfied . scipy.optimize gives you four specialized search engines:
minimize → make a scalar objective f ( x ) f(\mathbf{x}) f ( x ) small as possible.fsolve → find x \mathbf{x} x F ( x ) = 0 \mathbf{F}(\mathbf{x}) = \mathbf{0} F ( x ) = 0 curve_fit → choose parameters so a model matches data (least squares).linprog → optimize a linear objective under linear constraints.
They share one DNA: you hand them a function , a starting guess , and they iterate.
Intuition Why not just solve by hand?Real objectives are nonlinear, high-dimensional, and have no closed-form solution. You cannot set ∇ f = 0 \nabla f = 0 ∇ f = 0 iterate numerically : start somewhere, look at the local slope/curvature, step toward improvement, repeat until the step becomes negligible.
The whole family is built on one mathematical observation : at a minimum, the gradient is zero.
Definition Stationary pointx ∗ \mathbf{x}^* x ∗ f f f ∇ f ( x ∗ ) = 0 \nabla f(\mathbf{x}^*) = \mathbf{0} ∇ f ( x ∗ ) = 0 minimum additionally needs the Hessian H H H
This is why minimization and root-finding are secretly the same problem: minimizing f f f ∇ f \nabla f ∇ f Different APIs, same skeleton.
fsolve uses a multidimensional version (the Jacobian J J J g ′ g' g ′ Δ = − J − 1 F \Delta = -J^{-1}\mathbf{F} Δ = − J − 1 F minimize methods like 'Newton-CG', 'BFGS' use the Hessian (or an approximation of it).
Definition Standard form linear programmin x c ⊤ x s.t. A u b x ≤ b u b , A e q x = b e q , l ≤ x ≤ u \min_{\mathbf{x}}\; \mathbf{c}^\top \mathbf{x}\quad\text{s.t.}\quad A_{ub}\mathbf{x}\le \mathbf{b}_{ub},\;\; A_{eq}\mathbf{x}= \mathbf{b}_{eq},\;\; \mathbf{l}\le\mathbf{x}\le\mathbf{u} min x c ⊤ x s.t. A u b x ≤ b u b , A e q x = b e q , l ≤ x ≤ u WHY a separate tool: with everything linear, the optimum always sits at a corner (vertex) of the feasible polytope. linprog exploits this with simplex/interior-point — far faster and globally optimal, no starting guess needed.
A linear objective has constant gradient — it never "flattens out" inside the region. So you keep sliding downhill along the constraint walls until you can't anymore: a vertex.
Worked example 1 — minimize a 2D bowlMinimize f ( x , y ) = ( x − 3 ) 2 + ( y + 1 ) 2 f(x,y) = (x-3)^2 + (y+1)^2 f ( x , y ) = ( x − 3 ) 2 + ( y + 1 ) 2 ( 3 , − 1 ) (3,-1) ( 3 , − 1 )
from scipy.optimize import minimize f = lambda v: (v[ 0 ] - 3 ) ** 2 + (v[ 1 ] + 1 ) ** 2 res = minimize(f, x0 = [ 0 , 0 ]) # x0 = starting guess print (res.x) # ≈ [ 3., -1.] print (res.fun) # ≈ 0 Why x0? Iterative methods need a seed. Why res.x not res? res is an OptimizeResult object; .x holds the solution, .fun the minimum value, .success the convergence flag.
Worked example 2 — fsolve a nonlinear systemSolve x 2 + y 2 = 25 x^2 + y^2 = 25 x 2 + y 2 = 25 x − y = 1 x - y = 1 x − y = 1
from scipy.optimize import fsolve def F (v): x, y = v return [x ** 2 + y ** 2 - 25 , x - y - 1 ] # each eqn = 0 sol = fsolve(F, x0 = [ 1 , 1 ]) print (sol) # ≈ [4., 3.] Why write -25 and -1? fsolve finds where the returned vector equals zero , so you move everything to one side. Why does x0 matter? A different seed (e.g. [-5,-5]) converges to the other root ( − 3 , − 4 ) (-3,-4) ( − 3 , − 4 )
Worked example 3 — curve_fit an exponentialFit y = a e − b x y = a\,e^{-b x} y = a e − b x
import numpy as np from scipy.optimize import curve_fit model = lambda x, a, b: a * np.exp( - b * x) x = np.linspace( 0 , 4 , 50 ) y = 2.5 * np.exp( - 1.3 * x) + 0.05 * np.random.randn( 50 ) popt, pcov = curve_fit(model, x, y, p0 = [ 1 , 1 ]) print (popt) # ≈ [2.5, 1.3] print (np.sqrt(np.diag(pcov))) # 1σ uncertainty on a,b Why p0? For nonlinear models a starting guess prevents divergence. Why pcov? Its diagonal gives parameter variances — free error bars.
Worked example 4 — linprog allocationMaximize profit 40 x 1 + 30 x 2 40x_1 + 30x_2 40 x 1 + 30 x 2 x 1 + x 2 ≤ 40 x_1+x_2\le 40 x 1 + x 2 ≤ 40 2 x 1 + x 2 ≤ 60 2x_1+x_2\le 60 2 x 1 + x 2 ≤ 60 x ≥ 0 x\ge 0 x ≥ 0
from scipy.optimize import linprog # linprog MINIMIZES, so negate c to maximize c = [ - 40 , - 30 ] A_ub = [[ 1 , 1 ],[ 2 , 1 ]] b_ub = [ 40 , 60 ] res = linprog(c, A_ub = A_ub, b_ub = b_ub, bounds = [( 0 , None ),( 0 , None )]) print (res.x) # ≈ [20., 20.] print ( - res.fun) # profit = 1400 Why negate c? linprog only minimizes; maximizing c ⊤ x c^\top x c ⊤ x − c ⊤ x -c^\top x − c ⊤ x Why bounds? Encodes x ≥ 0 x\ge 0 x ≥ 0 [ 0 , ∞ ) [0,\infty) [ 0 , ∞ )
Common mistake "fsolve found a wrong/no solution — the function is broken."Why it feels right: the math is correct, so the code should find the answer.
The truth: fsolve is local — it converges to the root nearest x0, or diverges. Fix: try several starting points; check res, info, ier, msg = fsolve(F, x0, full_output=True) and inspect ier==1.
Common mistake Forgetting that linprog minimizes.Why it feels right: "optimize" sounds symmetric. Truth: it always minimizes. Maximizing without negating c gives the worst plan. Fix: maximize f f f c = -f_coeffs, and report -res.fun.
Common mistake curve_fit with nop0 on a nonlinear model.
Why it feels right: linear np.polyfit needs no guess. Truth: nonlinear least squares is non-convex; bad default seeds → garbage or runtime error. Fix: supply a physically sensible p0.
Common mistake Confusing equations and objectives.Passing a system of equations to minimize (it would minimize their sum , not zero them) or passing a scalar cost to fsolve. Fix: roots → fsolve; smallest value → minimize.
Recall Feynman: explain to a 12-year-oldImagine you're blindfolded on a hilly field and want the lowest point. You feel which way the ground tilts and take a step downhill, again and again — that's minimize . If instead you want the spot where the ground is exactly flat at sea level , that's fsolve finding a zero. curve_fit is like bending a stretchy string to pass through scattered dots as snugly as possible. linprog is a city with straight streets and a "lowest" corner — you just walk along the walls to the best corner. Same idea every time: keep stepping until you can't improve.
Mnemonic Pick the right tool"My Friend Can't Lie" →
M inimize (smallest value) · F solve (find zeros) · C urve_fit (fit data) · L inprog (linear corners).
What does scipy.optimize.minimize minimize? A scalar objective
f ( x ) f(\mathbf{x}) f ( x ) , returning the
x \mathbf{x} x giving the smallest value.
Why are minimization and root-finding the same problem? Minimizing
f f f = finding a root of its gradient
∇ f = 0 \nabla f = 0 ∇ f = 0 .
Derive the Newton update for g ( x ) = 0 g(x)=0 g ( x ) = 0 Linearize
g ( x k + Δ ) ≈ g ( x k ) + g ′ ( x k ) Δ = 0 ⇒ x k + 1 = x k − g ( x k ) / g ′ ( x k ) g(x_k+\Delta)\approx g(x_k)+g'(x_k)\Delta=0 \Rightarrow x_{k+1}=x_k - g(x_k)/g'(x_k) g ( x k + Δ ) ≈ g ( x k ) + g ′ ( x k ) Δ = 0 ⇒ x k + 1 = x k − g ( x k ) / g ′ ( x k ) .
Why does curve_fit square the residuals? Squares keep errors positive (no cancellation) and give the maximum-likelihood fit under Gaussian noise.
What must you change to MAXIMIZE with linprog? Negate the cost vector c (and negate res.fun for the true value); linprog only minimizes.
Why does an LP optimum sit at a vertex? A linear objective has constant gradient, so it improves until it hits a corner of the feasible polytope.
What attribute of an OptimizeResult holds the solution vs the value? .x is the solution, .fun is the objective value, .success the convergence flag.
Why does fsolve sometimes give the "wrong" root? It's a local method — it converges to the root nearest x0; change the seed to find others.
What does the diagonal of pcov from curve_fit give you? Variances of the fitted parameters; sqrt(diag(pcov)) are 1σ uncertainties.
When use fsolve vs minimize for a system of equations? fsolve to make the system equal zero; minimize only if you genuinely want the smallest value of one scalar.
Newton's Method — the iterative core shared by minimize & fsolveGradient and Hessian — convergence conditions (∇ f = 0 \nabla f=0 ∇ f = 0 H ⪰ 0 H\succeq 0 H ⪰ 0 Least Squares Regression — theory behind curve_fitLinear Programming Simplex — algorithm inside linprogNumPy polyfit — closed-form linear-fit alternativeConvex Optimization — when a local min is the global min
minimizing f = root of grad f
minimized by Levenberg-Marquardt
Stationary point grad f = 0
minimize scalar objective
Least-squares objective S theta
Intuition Hinglish mein samjhoDekho, scipy.optimize ka funda simple hai: har problem mein hum kisi number space mein search kar rahe hote hain jab tak ek condition satisfy na ho jaye. minimize kisi function ki value ko sabse chhoti karta hai (jaise blindfold pahaad par sabse neeche jaana). fsolve woh point dhoondhta hai jahan equation zero ho jaye — yaani root finding. curve_fit noisy data ke through ek model curve fit karta hai (least squares se), aur linprog linear cost ko linear constraints ke under optimize karta hai.
Sabse pyaari baat: minimize aur fsolve andar se ek hi cheez hain. Minimum par gradient zero hota hai, to "minimize f f f f ′ = 0 f'=0 f ′ = 0 Newton's method hai — current guess par Taylor expansion karke linear part rakho, usse zero set karke step nikaalo: x k + 1 = x k − g / g ′ x_{k+1}=x_k - g/g' x k + 1 = x k − g / g ′ x0 (starting guess) dena padta hai aur galat guess se galat ya koi root nahi milta.
Practical tips jo exam aur projects mein bachayenge: linprog hamesha minimize karta hai, to maximize karna ho to c ko negate karo aur answer -res.fun lo. Nonlinear curve_fit mein p0 zaroor do warna garbage aata hai. Result object mein .x solution hai, .fun value, aur .success convergence batata hai. fsolve ka result confirm karne ke liye full_output=True se ier==1 check karo. Bas itna yaad rakho — "My Friend Can't Lie" = Minimize, Fsolve, Curve_fit, Linprog — aur tum set ho.
