Visual walkthrough — Solving nonlinear systems — Newton's method in n dimensions

We will build the whole method for the case a smart 12-year-old can see: two equations, two unknowns ($n=2$). Everything generalises to $n$ dimensions by replacing "$2$" with "$n$" — we point out exactly where.

Step 1 — What does "$\mathbf{F}(\mathbf{x})=0$" even look like?

WHAT. We have two knobs to turn — call them $x$ and $y$. We bundle them into one arrow (a point) called $\mathbf{x}=(x,y)$. Two rules must both come out to zero at the same time:

Each rule is a curve in the flat $(x,y)$ plane: the set of points where that rule reads zero. A root ${\mathbf{x}}^{\ast }$ is where the two curves cross.

WHY. Before touching any calculus, we must know the target: not "make one thing zero" but "make several things zero simultaneously" — a crossing point. Everything downstream is a machine for hunting that crossing.

PICTURE.

Step 2 — Zoom in until the curve looks straight

WHAT. Pick a guess ${\mathbf{x}}_k=(x_k,y_k)$. It is not on the crossing. Now zoom in hard on one rule's curve near ${\mathbf{x}}_k$. Any smooth curve, seen close enough, looks like a straight line — its tangent.

WHY. Straight things we can solve exactly (that is just algebra, no guessing). Curved things we cannot. So the whole trick is: replace the curve by its tangent line near the guess, solve the easy straight problem, and accept a small error because we're only pretending locally.

PICTURE.

Step 3 — Measure the slope in every direction: the Jacobian

WHAT. A curve in the plane has a slope, but $f_1$ actually depends on two knobs, so it has two slopes at ${\mathbf{x}}_k$: how fast it changes if we nudge $x$ (holding $y$ fixed), and how fast if we nudge $y$. Those two numbers are the partial derivatives $\frac{\partial f_1}{\partial x}$ and $\frac{\partial f_1}{\partial y}$.

Do this for both rules and stack the four numbers into a box — the Jacobian:

Row $i$ is "how rule $i$ tilts". Column $j$ is "what happens if I push knob $j$".

WHY. In 1-D, one slope number $f^{\prime }$ told us how to slide to the root. In 2-D there are two rules and two knobs, so the "slope" needs $2\times 2=4$ numbers. The Jacobian is exactly the $n$-D upgrade of $f^{\prime }$ — the Jacobian matrix.

PICTURE.

Step 4 — Write the flat ramp: the linear model

WHAT. Let $\mathbf{h}=(h_x,h_y)$ be a small step away from the guess. Multivariable Taylor's theorem (multivariable) says each rule, near ${\mathbf{x}}_k$, equals its value plus its slopes times the step, plus a tiny leftover:

Stack both rules and the "slope × step" part becomes exactly the Jacobian times the step:

WHY. We drop the tiny leftover $O(\Vert \mathbf{h}{\Vert }^2)$ on purpose — that is the only approximation in all of Newton's method. What remains is a flat ramp (a linear map): it is straight, so we can solve it exactly. The size of what we threw away is $\Vert \mathbf{h}{\Vert }^2$; that number will explain the blazing speed in Step 8.

PICTURE.

Step 5 — Demand the ramp hits zero, and solve

WHAT. We want the true $\mathbf{F}$ to be zero. We can't force that on the curve, but we can force it on our flat ramp. Set the model to $0$ and solve for the step $\mathbf{h}$:

Reading the box: "the Jacobian, applied to the unknown step $\mathbf{h}$, must undo the current miss $\mathbf{F}({\mathbf{x}}_k)$." This is a plain linear system $A\mathbf{h}=\mathbf{b}$ with $A=J$, $\mathbf{b}=-\mathbf{F}$ — the exact kind Gaussian elimination and LU decomposition eat for breakfast.

WHY. This step is the payoff of all the zooming: we converted an impossible nonlinear demand into a solvable straight one. We solve it (never invert $J$ — see mistake box), rename the answer $\Delta {\mathbf{x}}_k=\mathbf{h}$, and step:

PICTURE.

Step 6 — Watch one full iteration land closer

WHAT. With the step taken, ${\mathbf{x}}_{k+1}$ sits where the tangent crossed zero. The real curve wasn't perfectly straight, so we're not exactly on the root — but we're much closer. Re-measure slopes there, build a fresh ramp, step again.

WHY. One step of "zoom → flatten → solve → step" is the entire algorithm. Repeating it walks us onto the crossing. Watch how the miss $\Vert \mathbf{F}\Vert$ collapses.

PICTURE.

Step 7 — The degenerate case: a flat or crossing-slope Jacobian

WHAT. The ramp only has one place where it hits zero if it actually tilts. If $J({\mathbf{x}}_k)$ is singular ($\det J=0$), the two rows are parallel — the ramp is a valley or a flat sheet, not a single point. Then $J\Delta \mathbf{x}=-\mathbf{F}$ has no solution or infinitely many: the step is undefined or shoots to infinity.

WHY. This is the exact twin of scalar Newton dividing by $f^{\prime }=0$ at a flat spot. It happens (a) when the two curves touch tangentially instead of crossing cleanly (a double root), or (b) mid-iteration when a guess lands on a bad ridge. Cures: re-choose ${\mathbf{x}}_0$, or use damped Newton (take a fraction of the step) — see Convergence order of iterative methods and Broyden's method for the practical fixes. When $J({\mathbf{x}}^{\ast })$ is singular at the root, the quadratic speed of Step 8 downgrades to slow crawling.

PICTURE.

Step 8 — Why the error squares every step

WHAT. Let the error be ${\mathbf{e}}_k={\mathbf{x}}_k-{\mathbf{x}}^{\ast }$ (how far the guess is from the true crossing). The only thing we ever threw away was the $O(\Vert \mathbf{h}{\Vert }^2)$ leftover of Step 4. That leftover is the new error:

If today's error is $10^{-2}$, tomorrow's is $\sim 10^{-4}$, then $10^{-8}$, then $10^{-16}$ — the number of correct digits doubles each step.

WHY. This "square the error" behaviour is called quadratic convergence (see Convergence order of iterative methods). It holds when (1) ${\mathbf{x}}_0$ is close enough and (2) $J({\mathbf{x}}^{\ast })$ is nonsingular (Step 7). It is why Newton beats slow methods like Fixed-point iteration, whose error only shrinks by a constant factor each step (linear convergence).

PICTURE.

The one-picture summary

Everything at once: the true curve (cyan), the guess ${\mathbf{x}}_k$, the flat tangent ramp (white), the step $\Delta \mathbf{x}$ that lands where the ramp hits zero, and the shrinking hop toward the root ${\mathbf{x}}^{\ast }$ (amber). Solve $J\Delta \mathbf{x}=-\mathbf{F}$, add, repeat.

Quick self-check