Convex vs non-convex functions: Visual walkthrough

We will earn every symbol before using it. By the last figure you will see why non-convex landscapes have "fake bottoms" and convex ones don't.

Step 0 — What is a "graph", a "chord", and a "tangent"?

Before any formula, three plain-English pictures.

A graph of a function $f$ is just the set of dots you get by sweeping an input $x$ left-to-right and stacking its output $f (x)$ upward as height. Think of a landscape seen from the side: horizontal = where you stand, vertical = how high the ground is.
A chord is a straight ruler laid across the graph touching it at exactly two points. It is the shortcut between two spots on the curve.
A tangent is a straight ruler that just kisses the graph at one point without crossing it — the direction you'd slide if you were a marble sitting right there.

Step 1 — The convex inequality, symbol by symbol

WHAT. We write down what "curve stays below the chord" means as a formula.

WHY. A picture is convincing but not checkable. We need something a machine — or a proof — can test. So we translate the picture into arithmetic.

First we need to name the "in-between point." Slide a knob $λ$ (Greek lambda) from $0$ to $1$ . It is a mixing dial:

$m (λ) = weight on x λ x + weight on y (1 - λ) y$

When $λ = 1$ : $m = x$ (all weight on $x$ ).
When $λ = 0$ : $m = y$ (all weight on $y$ ).
When $λ = \frac{1}{2}$ : $m = \frac{x + y}{2}$ , the exact midpoint.

So $λ$ walks us smoothly along the horizontal axis from $y$ to $x$ .

Now the definition:

$f (curve height at the mixed input λ x + (1 - λ) y) \leq ruler height above that same input λ f (x) + (1 - λ) f (y)$

Left side: go to the mixed input $m$ , then read the curve's height — the black dot.
Right side: the same mixing dial applied to the two heights $f (x)$ and $f (y)$ — that is exactly where the red ruler sits above $m$ .
$\leq$ : curve height is at most ruler height. Curve below chord. That is convex.

Step 2 — The tangent test (the "instant chord")

WHAT. We replace the two-point chord with a one-point tangent line.

WHY. Chords compare two far-apart points. Optimization asks a local question: "from where I stand, where is down?" That is a tangent question. So we shrink one end of the chord toward the other.

To talk about a tangent we need the word slope: the slope of $f$ at $x$ is how fast height changes per step sideways — steepness. In one dimension we write it $f^{'} (x)$ (read "f-prime of x"). The tangent line through the point $(x, f (x))$ is:

$tangent height at y = start height f (x) + slope f^{'} (x) sideways step (y - x)$

$f (x)$ : where the tangent is pinned.
$f^{'} (x) (y - x)$ : rise = slope $\times$ run, the straight-line prediction of height at $y$ .

The tangent version of convex says the curve sits above every tangent:

$f (y) \geq f (x) + f^{'} (x) (y - x)$

Step 3 — The flat-spot punchline (why convex ⇒ global min)

WHAT. We plug a flat tangent into Step 2.

WHY. Gradient Descent stops where the slope is zero — a stationary point. We want to know what a zero slope guarantees for a convex function.

Suppose at some point $x^{⋆}$ the slope is zero: $f^{'} (x^{⋆}) = 0$ . Put that into the boxed tangent inequality:

$f (y) \geq f (x^{⋆}) + = 0 f^{'} (x^{⋆}) (y - x^{⋆}) = f (x^{⋆})$

The whole slope term vanishes because $f^{'} (x^{⋆}) = 0$ .
What remains: $f (y) \geq f (x^{⋆})$ for every $y$ .

Read that last line out loud: no point anywhere is lower than $x^{⋆}$ . A flat spot on a convex curve is the global minimum — not merely local. There is no "over the hill" that could be deeper, because the flat tangent is a floor the whole graph rests on.

Step 4 — Where does convexity come from? Curvature $f^{''} \geq 0$

WHAT. We show a curve guarantees the tangent test if it never bends downward.

WHY. Steps 2–3 assumed convexity. But how do we check it on a real function? We need a computable handle. That handle is the second derivative $f^{''} (x)$ — the slope of the slope, i.e. how fast the steepness itself changes. If $f^{''} \geq 0$ , the steepness only ever increases: the curve keeps turning upward, cupping like a bowl.

We use one tool — Taylor's theorem with exact remainder — because it is the only device that writes a curve's value exactly in terms of a tangent line plus a leftover curvature term:

$f (y) = the tangent line (Step 2) f (x) + f^{'} (x) (y - x) + curvature leftover, some c between x, y \frac{1}{2} f^{''} (c) (y - x)^{2}$

The first bracket is exactly the tangent height from Step 2.
The second bracket is everything the tangent missed, packed into a single curvature term evaluated at some hidden point $c$ sitting between $x$ and $y$ .

Now bound the leftover. Two of its pieces are always $\geq 0$ :

$\frac{1}{2} > 0, (y - x)^{2} \geq 0 (a square), f^{''} (c) \geq 0 (our assumption) .$

Multiply three non-negative things → the leftover is $\geq 0$ . Drop it, and the equation becomes an inequality:

$f (y) \geq f (x) + f^{'} (x) (y - x) .$

That is the boxed tangent test of Step 2. So $f^{''} \geq 0$ everywhere forces convexity. Chain the steps: $f^{''} \geq 0 \Rightarrow$ tangent test $\Rightarrow$ flat spot is global min.

See Taylor Series Expansion for the machinery, and Hessian Matrix for the multi-dimensional twin of $f^{''}$ .

Step 5 — The degenerate case: $f^{''} = 0$ (a flat straight line)

WHAT. We test the boundary where curvature is exactly zero.

WHY. The rule was $f^{''} \geq 0$ , allowing equality. A good walkthrough must show the edge, not just the strict interior. What happens when the curve doesn't cup at all?

If $f^{''} (x) = 0$ everywhere, the leftover term is exactly $0$ , so Step 4 gives $f (y) = f (x) + f^{'} (x) (y - x)$ with equality. The "curve" is a straight line — an affine function. It sits on every chord and on every tangent simultaneously.

Curve $\leq$ chord? Yes (equal counts as $\leq$ ). So it is convex.
Curve $\geq$ chord? Also yes. So it is concave too.

A straight line is the one shape that is convex and concave at once. This is the razor's edge: nudge $f^{''}$ up a hair and you get a bowl; nudge it down and you get a dome.

Step 6 — The failure case: $f^{''}$ changes sign (non-convex, fake bottoms)

WHAT. We break the rule on purpose and watch the guarantee die.

WHY. The parent's whole warning — Gradient Descent can get trapped — lives here. We must see the trap.

Take $f (x) = x^{3} - x$ . Its curvature is $f^{''} (x) = 6 x$ , which is negative for $x < 0$ and positive for $x > 0$ . Where $f^{''} < 0$ the curve bends downward (domes), where $f^{''} > 0$ it cups. The curve threads below some chords and above others — the convex inequality fails.

The damage: there is a flat spot (a local minimum) that is not the global bottom. A marble released nearby settles there, "thinks" it is done, while a deeper valley waits over the ridge. The tangent floor of Step 3 no longer supports the whole graph because there is no single tangent everything sits above. This is exactly a saddle-like / multi-basin landscape, the reason Local vs Global Minima stop agreeing.

The one-picture summary

Everything above compressed into a single frame: a convex bowl (left) whose only flat spot is the true bottom, versus a non-convex ridge (right) with a fake bottom trapping the marble. The red marble tells the whole story — one honest valley vs. many deceptive dips.

Recall Feynman retelling — the whole walkthrough in plain words

Picture a skate ramp seen from the side. Lay a ruler across any two points on it. If the ramp always ducks under the ruler, it's a convex bowl. Now shrink the ruler until it just kisses one spot — that's a tangent, the way a marble would roll from there. Convex means the ramp never dips under any such kiss-line either; every tangent is a floor the ramp rests on. So if you find a spot where the tangent is perfectly flat, that flat floor lies under the entire ramp — nothing anywhere is lower. That flat spot is the deepest point, period. Why is a ramp convex in the first place? Because its steepness only ever increases as you walk right — it keeps curling upward ( $f^{''} \geq 0$ ). A perfectly straight ramp ( $f^{''} = 0$ ) is the borderline: bowl and dome at once. But bend a piece downward somewhere ( $f^{''} < 0$ ) and you've built a skate park with little dips; a marble parks in the nearest one and swears it's done, while a deeper valley hides over the hill. Convex = one honest bowl. Non-convex = fake bottoms.

Connections

Gradient Descent — Step 3 is exactly why it can be trusted on convex losses.
Hessian Matrix — the multi-dimensional $f^{''}$ ; needs Positive Semidefinite Matrices.
Taylor Series Expansion — the engine of Step 4.
Saddle Points · Local vs Global Minima · Loss Functions · Convex Optimization.