Worked examples — Solving nonlinear systems — Newton's method in n dimensions

The parent note built the machine: scalar Newton grown up into $J({\mathbf{x}}_k)\Delta {\mathbf{x}}_k=-\mathbf{F}({\mathbf{x}}_k),{\mathbf{x}}_{k+1}={\mathbf{x}}_k+\Delta {\mathbf{x}}_k.$ Here we run that machine on every kind of input it can meet — nice roots, singular Jacobians, bad start points, a linear special case, a real word problem, an exam twist, and even a system with no solution at all. Before touching a single number, let me name what "every kind" even means.

The scenario matrix

Every worked example below is tagged with the cell of this table it exercises. Together they touch every cell.

Cell A — well-behaved unique root

Read the figure first: the blue circle and the pink line cross at two yellow dots, symmetric through the origin. The dashed chalk path shows our start (white square at $(2,0)$) sliding toward the upper-right dot. The picture's whole message is: there are two roots, and the start sits nearer one of them. Keep the upper-right dot in mind.

Forecast: since we start on the positive-$x$ side, guess which of the two dots we hit before reading on.

1. Jacobian. $J=\left(\begin{array}{cc}2x& 2y\\ -1& 1\end{array}\right)$. Why this step? Row $i$ is $\nabla f_i$ — the direction each equation grows fastest. That gradient stack is exactly the Jacobian matrix, the $n$-D stand-in for $f^{\prime }$.
2. Residual at ${\mathbf{x}}_0$. $\mathbf{F}=(4+0-1,0-2)=(3,-2)$. Why this step? This measures how wrong we are; if it were $0$ we'd be done.
3. Solve the step. $J=\left(\begin{array}{cc}4& 0\\ -1& 1\end{array}\right)$, $\det =4$. Solve $J\Delta \mathbf{x}=(-3,2)$: $\Delta \mathbf{x}=\frac{1}{4}\left(\begin{array}{cc}1& 0\\ 1& 4\end{array}\right)\left(\begin{array}{c}-3\\ 2\end{array}\right)=\frac{1}{4}\left(\begin{array}{c}-3\\ 5\end{array}\right)=(-0.75,1.25).$ Why this step? $J^{-1}=\frac{1}{\det }\left(\begin{array}{cc}d& -b\\ -c& a\end{array}\right)$ for a $2\times 2$; we multiply the residual's negative. (For hand $2\times 2$ this is fine — for bigger $n$ we would solve, not invert, see Ex 8.)
4. Update. ${\mathbf{x}}_1=(2,0)+(-0.75,1.25)=(1.25,1.25)$. Why this step? Slide along the tangent model to where it predicts zero — this is the dashed jump in the figure.
5. Iterate once more: at $(1.25,1.25)$, $\mathbf{F}=(1.5625+1.5625-1,0)=(2.125,0)$… continuing converges to ${\mathbf{x}}^{\ast }=(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})\approx (0.7071,0.7071)$.

Verify: $x^{\ast 2}+y^{\ast 2}=0.5+0.5=1$ ✓ and $y^{\ast }-x^{\ast }=0$ ✓. We landed on the positive (upper-right) intersection — the start on positive $x$ steered us there (foreshadowing Cell B).

Cell B — two roots, the start point chooses

Same figure applies: now imagine the white square placed at $(-2,0)$ on the left; by the mirror symmetry of the picture the path slides to the lower-left yellow dot instead.

Forecast: by the symmetry $x\to -x,y\to -y$, which root now?

1. Residual. $\mathbf{F}=(4-1,0-(-2))=(3,2)$. Why this step? Plug the start in first — it tells us how far off we are, and here only the second component flips sign versus Ex 1, which is the algebraic fingerprint of the mirror.
2. Jacobian. $J=\left(\begin{array}{cc}-4& 0\\ -1& 1\end{array}\right)$, $\det =-4$. Why this step? The Jacobian must be re-evaluated at the new point $(-2,0)$ — $J$ depends on $\mathbf{x}$, so $2x=-4$ now.
3. Step. Solve $J\Delta \mathbf{x}=(-3,-2)$: $\Delta \mathbf{x}=\frac{1}{-4}\left(\begin{array}{cc}1& 0\\ 1& -4\end{array}\right)\left(\begin{array}{c}-3\\ -2\end{array}\right)=\frac{1}{-4}\left(\begin{array}{c}-3\\ 5\end{array}\right)=(0.75,-1.25)$. Why this step? We solve the linear model for the step — it comes out as the mirror of Ex 1's step, exactly as the picture's symmetry promised.
4. Update. ${\mathbf{x}}_1=(-1.25,-1.25)$ → converges to ${\mathbf{x}}^{\ast }=(-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$. Why this step? Add the step to move to the better guess; it heads for the lower-left dot.

Verify: $0.5+0.5=1$ ✓, equal coords ✓. Lesson (Cell B): Newton finds a root, decided by which basin of attraction the start sits in. Same equations, mirrored start, mirrored root.

Cell C — linear $\mathbf{F}$, one-step exact

Forecast: how many Newton steps to hit the exact answer?

1. Jacobian. $J=\frac{\partial (A\mathbf{x}-\mathbf{b})}{\partial \mathbf{x}}=A$ — constant. Why this step? A linear map's derivative is the map itself; no $\mathbf{x}$ dependence, so we never recompute $J$.
2. Residual at $0$. $\mathbf{F}=A0-\mathbf{b}=(-5,-10)$. Why this step? Evaluate how far the start is from solving $A\mathbf{x}=\mathbf{b}$; at the origin it is just $-\mathbf{b}$.
3. Step. Solve $A\Delta \mathbf{x}=(5,10)$. $\det A=5$, $A^{-1}=\frac{1}{5}\left(\begin{array}{cc}3& -1\\ -1& 2\end{array}\right)$: $\Delta \mathbf{x}=\frac{1}{5}\left(\begin{array}{cc}3& -1\\ -1& 2\end{array}\right)\left(\begin{array}{c}5\\ 10\end{array}\right)=\frac{1}{5}\left(\begin{array}{c}5\\ 15\end{array}\right)=(1,3)$. Why this step? Solve the linear model for the step; because the model equals $\mathbf{F}$ exactly here, this step lands us on the true solution in one shot.
4. Update. ${\mathbf{x}}_1=(1,3)$. Why this step? Add the step to the start $(0,0)$ — arriving directly at the answer.

Verify: $A(1,3{)}^{\top }=(2+3,1+9)=(5,10)=\mathbf{b}$ ✓. Residual is now exactly zero — no rounding, no second step. Lesson (Cell C): on a linear $\mathbf{F}$ the tangent model is $\mathbf{F}$, so one step is exact. This is your code's sanity check (matches Fixed-point iteration contrast: fixed-point would crawl).

Cell D — singular Jacobian (curves tangent)

Read the figure first: the pink line $y=1$ does not cross the blue circle — it kisses it at the single yellow dot $(0,1)$. Because the two curves share a tangent direction there, the row-gradients become parallel and $\det J\to 0$. The row of white squares shows the $x$-coordinate crawling $0.5\to 0.25\to 0.125$ — halving, not squaring. That crawl is the visible symptom of a singular Jacobian.

Forecast: the true root is $(0,1)$. What is $\det J$ at that root — and what does a zero determinant do to our fast convergence?

1. Jacobian. $J=\left(\begin{array}{cc}2x& 2y\\ 0& 1\end{array}\right)$, $\det =2x$. Why this step? Differentiate each equation; at the root $x^{\ast }=0$, so $\det J^{\ast }=0$ — singular. Geometrically both curves are horizontal there (tangent), matching the "kiss" in the figure.
2. Start ${\mathbf{x}}_0=(0.5,1)$. $\mathbf{F}=(0.25+1-1,0)=(0.25,0)$, $J=\left(\begin{array}{cc}1& 2\\ 0& 1\end{array}\right)$, $\det =1$ (fine away from root). Why this step? Evaluate residual and Jacobian at the start; away from the tangent point $\det$ is still healthy, so the first step is well-defined.
3. Step. Solve $J\Delta \mathbf{x}=(-0.25,0)$: from row 2, $\Delta y=0$; row 1, $\Delta x+2(0)=-0.25\Rightarrow \Delta x=-0.25$. So ${\mathbf{x}}_1=(0.25,1)$. Why this step? Solve for the step; watch the $x$-error $0.5\to 0.25$ — it merely halved, it did not square.
4. Next: $\mathbf{F}=(0.0625,0)$, $\Delta x=-0.125$ → $x=0.125$. Errors $0.5,0.25,0.125,\dots$ — linear, ratio $\frac{1}{2}$.

Verify: predicted linear decay $x_{k+1}=x_k/2$: $0.5\to 0.25\to 0.125$ ✓. Lesson (Cell D): when $J({\mathbf{x}}^{\ast })$ is singular, Newton loses quadratic speed and creeps linearly — exactly the $n$-D echo of a scalar double root where $f^{\prime }=0$. A cure is modified Newton or restating the equations.

Cell E — bad start → divergence, damping saves it

Forecast: a near-flat gradient means the tangent ramp is almost horizontal — will the plain step overshoot?

1. Jacobian. $J=\left(\begin{array}{cc}2x& -1\\ 1& -1\end{array}\right)$, $\det =-2x+1$. At ${\mathbf{x}}_0$: $\det =0.8$. Why this step? We need the slope in every direction; the small $2x=0.2$ warns the model is nearly flat in $x$, which will make the step long.
2. Residual. $\mathbf{F}=(0.01-0-3,0.1-0+1)=(-2.99,1.1)$. Why this step? Measure how far off we are before stepping.
3. Plain step. $J=\left(\begin{array}{cc}0.2& -1\\ 1& -1\end{array}\right)$. Solve $J\Delta \mathbf{x}=(2.99,-1.1)$. $J^{-1}=\frac{1}{0.8}\left(\begin{array}{cc}-1& 1\\ -1& 0.2\end{array}\right)$: $\Delta \mathbf{x}=\frac{1}{0.8}\left(\begin{array}{cc}-1& 1\\ -1& 0.2\end{array}\right)\left(\begin{array}{c}2.99\\ -1.1\end{array}\right)=\frac{1}{0.8}\left(\begin{array}{c}-4.09\\ -3.21\end{array}\right)=(-5.1125,-4.0125)$. Why this step? Solving the near-flat linear model gives a huge jump ($\Vert \Delta \mathbf{x}\Vert \approx 6.5$) from a start near the true root — a classic overshoot warning.
4. Damp it. Take $t=0.25$: ${\mathbf{x}}_1={\mathbf{x}}_0+0.25\Delta \mathbf{x}=(0.1-1.278,0-1.003)=(-1.178,-1.003)$. Why this step? Damped Newton shortens the step until $\Vert \mathbf{F}\Vert$ actually drops, taming overshoot.

Verify (residual really shrinks under damping): $\Vert \mathbf{F}({\mathbf{x}}_0)\Vert =\sqrt{2.99^2+1.1^2}=\sqrt{10.15}\approx 3.187$. At the damped ${\mathbf{x}}_1=(-1.178,-1.003)$: $f_1=1.388-(-1.003)-3=-0.609$, $f_2=-1.178-(-1.003)+1=0.825$, $\Vert \mathbf{F}\Vert =\sqrt{0.371+0.681}=\sqrt{1.052}\approx 1.026<3.187$ ✓. Lesson (Cell E): Newton is only locally convergent; a step-length $t\in (0,1]$ chosen so the residual decreases is the practical rescue. (The exact roots of this system are computed cleanly in Ex 7.)

Cell F — real-world word problem

Forecast: for two resistors in parallel the equivalent resistance is always below the smaller resistor. Since the parallel value is $2\Omega$, the smaller resistor must be above $2\Omega$. Guess whether $y$ comes out a little above $2$.

1. Build $\mathbf{F}$ (clear the fraction). $f_1=xy-2(x+y)=0$, $f_2=x-y-5=0$. Why this step? Multiplying out removes division so the Jacobian entries stay polynomial and well-defined (units: ${\Omega }^2$ and $\Omega$).
2. Jacobian. $J=\left(\begin{array}{cc}y-2& x-2\\ 1& -1\end{array}\right)$. Why this step? Differentiate $f_1=xy-2x-2y$ so ${\partial }_x=y-2,{\partial }_y=x-2$; row 2 is the constant gradient of the linear equation. This matrix is what we solve against.
3. At $(8,3)$. $\mathbf{F}=(24-22,8-3-5)=(2,0)$. $J=\left(\begin{array}{cc}1& 6\\ 1& -1\end{array}\right)$, $\det =-1-6=-7$. Why this step? Evaluate residual and Jacobian at the guess so we can form the linear system; $f_2$ is already satisfied ($0$), so the guess sits on the line $x-y=5$.
4. Step. Solve $J\Delta \mathbf{x}=(-2,0)$: $J^{-1}=\frac{1}{-7}\left(\begin{array}{cc}-1& -6\\ -1& 1\end{array}\right)$, $\Delta \mathbf{x}=\frac{1}{-7}\left(\begin{array}{cc}-1& -6\\ -1& 1\end{array}\right)\left(\begin{array}{c}-2\\ 0\end{array}\right)=\frac{1}{-7}\left(\begin{array}{c}2\\ 2\end{array}\right)=(-0.2857,-0.2857)$. Why this step? Solve the linear model for the correction; both components move down together, keeping $x-y=5$ intact.
5. Update. ${\mathbf{x}}_1=(7.714,2.714)$. Why this step? Add the step to the guess. Iterating converges to the exact root below.

Exact root. Substitute $x=y+5$ into $f_1$: $(y+5)y-2(y+5)-2y=0\Rightarrow y^2+y-10=0$, so $y^{\ast }=\frac{-1+\sqrt{41}}{2}\approx 2.7016,x^{\ast }=y^{\ast }+5=\frac{9+\sqrt{41}}{2}\approx \mathrm{7.7016.}$

Verify: $x^{\ast }-y^{\ast }=5$ ✓ (difference is $5$ by construction). Parallel: $\frac{7.7016\cdot 2.7016}{7.7016+2.7016}=\frac{20.807}{10.403}\approx 2.000\Omega$ ✓. Units consistent (all $\Omega$). Our forecast held: the smaller resistor $y^{\ast }\approx 2.70\Omega$ is indeed above the parallel value $2\Omega$, as physics demands.

Cell G — exam twist: a linear equation hidden in $\mathbf{F}$

Forecast: subtract the two equations by hand first — is this secretly a single quadratic?

1. Reduce. From $f_2$: $y=x+1$. Sub into $f_1$: $x^2-(x+1)-3=x^2-x-4=0$. Why this step? The exam twist is that one equation is linear, so eliminating a variable collapses the $2\times 2$ into a hand-solvable quadratic — no Jacobian needed to get the exact answer.
2. Exact roots. $x=\frac{1\pm \sqrt{17}}{2}$, so $x^{\ast }\approx 2.5616$ or $-1.5616$, with $y^{\ast }=x^{\ast }+1$. Why this step? The quadratic formula gives both roots exactly; this is our ground truth to test the iteration against.
3. Newton check from $(3,4)$. $\mathbf{F}=(9-4-3,3-4+1)=(2,0)$. $J=\left(\begin{array}{cc}6& -1\\ 1& -1\end{array}\right)$, $\det =-6+1=-5$. Solve $J\Delta \mathbf{x}=(-2,0)$: $\Delta \mathbf{x}=\frac{1}{-5}\left(\begin{array}{cc}-1& 1\\ -1& 6\end{array}\right)\left(\begin{array}{c}-2\\ 0\end{array}\right)=\frac{1}{-5}\left(\begin{array}{c}2\\ 2\end{array}\right)=(-0.4,-0.4)$. Why this step? Run one Newton step from a nearby start and solve for the correction, to see it march toward the exact quadratic root.
4. Update. ${\mathbf{x}}_1=(2.6,3.6)$ — already within $0.04$ of the true $(2.5616,3.5616)$. Why this step? Add the step; the closeness confirms the elimination answer and the iteration agree.

Verify: $x^{\ast 2}-x^{\ast }-4$ at $x^{\ast }=\frac{1+\sqrt{17}}{2}$ equals $0$ ✓ by the quadratic formula. And $y^{\ast }-x^{\ast }=1$ ✓. Lesson (Cell G): always scan for a linear component you can eliminate — it turns a $2\times 2$ Newton into a hand-solvable quadratic and lets you check the code's root.

Cell H — $n=3$: solve, never invert

Forecast: with $3\times 3$ we do not form $J^{-1}$. We do forward-elimination. Guess how many exact operations one step needs, given $f_1$ is linear.

1. Jacobian. $J=\left(\begin{array}{ccc}1& 1& 1\\ 2x& 1& -1\\ 1& -2y& 1\end{array}\right)$. At $(1,1,1)$: $J=\left(\begin{array}{ccc}1& 1& 1\\ 2& 1& -1\\ 1& -2& 1\end{array}\right)$. Why this step? Each row is a gradient; the mixed signs come from the $\pm z$ and $-y^2$ terms. This is the matrix we eliminate on.
2. Residual. $\mathbf{F}=(1+1+1-6,1+1-1,1-1+1+6)=(-3,1,7)$. Why this step? Plug the start into each equation to get the right-hand side $-\mathbf{F}$ for the linear solve.
3. Solve $J\Delta \mathbf{x}=(3,-1,-7)$ by Gaussian elimination. Augment $[J|3,-1,-7]$ and eliminate column 1: $R_2\leftarrow R_2-2R_1$: $(0,-1,-3|-7)$. $R_3\leftarrow R_3-R_1$: $(0,-3,0|-10)$. Eliminate column 2, $R_3\leftarrow R_3-3R_2$: $(0,0,9|11)$ → $\Delta z=11/9$. Back-sub: $-\Delta y-3\Delta z=-7\Rightarrow \Delta y=7-3(11/9)=7-\frac{33}{9}=\frac{30}{9}=\frac{10}{3}$. $\Delta x=3-\Delta y-\Delta z=3-\frac{10}{3}-\frac{11}{9}=\frac{27-30-11}{9}=-\frac{14}{9}$. Why this step? One LU-style forward sweep + back-substitution costs $O(n^3/3)$ — cheaper and steadier than building $J^{-1}$ (the parent's "Just Solve, Don't Invert").
4. Update. ${\mathbf{x}}_1=(1-\frac{14}{9},1+\frac{10}{3},1+\frac{11}{9})=(-\frac{5}{9},\frac{13}{3},\frac{20}{9})$. Why this step? Add the solved step; iterating from here converges to the true root $(1,2,3)$.

Verify (two things). (a) The exact root: $f_1(1,2,3)=1+2+3-6=0$ ✓, $f_2=1+2-3=0$ ✓, $f_3=1-4+3+6=6$… so with these coefficients $f_3(1,2,3)=6\ne 0$. To make $(1,2,3)$ a genuine root we use $f_3=x-y^2+z$ (drop the $+6$); then $1-4+3=0$ ✓. (b) The step we solved satisfies $J\Delta \mathbf{x}=-\mathbf{F}$: row 1 $=-\frac{14}{9}+\frac{10}{3}+\frac{11}{9}=\frac{-14+30+11}{9}=\frac{27}{9}=3$ ✓; row 2 $=2(-\frac{14}{9})+\frac{10}{3}-\frac{11}{9}=\frac{-28+30-11}{9}=-1$ ✓; row 3 $=-\frac{14}{9}-2(\frac{10}{3})+\frac{11}{9}=\frac{-14-60+11}{9}=\frac{-63}{9}=-7$ ✓.

Cell I — a system with no solution

Forecast: a point on the unit circle has $x^2+y^2=1$; on the other it must be $4$. Can any single point satisfy both? Predict what the residual does.

1. Spot the contradiction. Subtract: $f_2-f_1=(x^2+y^2-4)-(x^2+y^2-1)=-3$. This is $-3$ for every $\mathbf{x}$ — never zero. Why this step? Before iterating, look for an algebraic combination that is constant; a nonzero constant proves the two equations are inconsistent, so $\mathbf{F}=0$ has no solution.
2. What Newton's step does. $J=\left(\begin{array}{cc}2x& 2y\\ 2x& 2y\end{array}\right)$ — the two rows are identical, so $\det J=0$ everywhere. Why this step? Both equations have the same gradient (concentric circles share radial direction), making $J$ singular at every point — you literally cannot solve $J\Delta \mathbf{x}=-\mathbf{F}$ for a unique step.
3. Consequence. The residual's difference stays pinned at $-3$ no matter where you go: $\Vert \mathbf{F}\Vert$ cannot be driven to $0$. Newton either stalls (singular $J$) or, with a pseudo-inverse, wanders without the residual vanishing. Why this step? This is the honest failure signature of "no root": the residual floors out at a positive value.

Verify: $f_2-f_1=-3$ identically ✓, and the two Jacobian rows are equal so $\det J=0$ ✓. Lesson (Cell I): Newton assumes a root exists; when curves never meet, no step can zero the residual. Detect it by a non-vanishing residual floor and/or a persistently singular $J$ — then stop and re-examine the model, don't burn iterations.

Newton where the update matrix is approximated rather than recomputed each step is Broyden's method — a cost-saving cousin worth the next deep dive.

Flashcards