Foundations — Softmax for output layers

Before you can read the parent note, you need to genuinely understand every symbol it throws at you. This page builds them one at a time, from nothing, each picture earning the next.

0. The starting picture: a list of numbers

Everything begins with a list of numbers. In math we bundle a list into one object called a vector and write it in bold: $\mathbf{z}$. The plain-word meaning is "an ordered row of numbers, each in its own slot."

Why the topic needs this. A classifier does not output one number — it outputs one score per class. Cat, dog, bird → three slots. So we always work with a whole list $\mathbf{z}$, never a single value.

1. The subscript $i$ and the summation sign $\sum$

You will constantly see two indices, $i$ and $j$. They are just pointer letters that walk down the slots of the list. There is nothing special about the letter — $i$ and $j$ are two different pointers so we can talk about two slots at the same time.

Why the topic needs this. The softmax denominator is a sum over every slot: ${\sum }_{j=1}^K{e}^{z_j}$. Without the $\sum$ symbol we could not write "add up all the exponentiated scores" in one compact line.

2. What a probability distribution actually is

The parent note's goal is to output a probability distribution. Strip the fancy word away: it is just a list of pizza slices.

The picture: one full pizza cut into $K$ slices. A slice's size is $p_i$. A slice cannot be negative (you can't have negative pizza) and cannot be bigger than the whole pizza. All slices together rebuild the one whole pizza — that is the "sum to 1" rule.

Why the topic needs this. "The network is 66% sure it's a cat" only makes sense if the outputs obey these two rules. Softmax exists precisely to guarantee them.

3. The number $e$ and the exponential $e^z$

This is the heart symbol of softmax, so we build it carefully.

Look at the curve. On the far left ($z$ very negative) the output hugs zero but never touches it — that is why probabilities are strictly positive. As $z$ grows, the curve rockets upward — a logit only slightly ahead ends up much larger after exponentiating, which is why softmax produces a confident winner.

Why the topic needs this. Raw logits can be negative; you can't take a slice of negative size. $e^{z_i}$ launders every raw score into a positive "weight" while preserving the order.

4. Logits — the raw scores $z_i$

The word "logit" comes from log-odds — see Logits and Log-Odds. For now hold this picture: a logit is a loudness of a vote, not yet a probability. Softmax is the translator from loudness → pizza slice.

Why the topic needs this. Softmax's input is always logits. Feeding it already-normalized numbers (Mistake 3 in the parent) breaks the intended meaning.

5. Putting the symbols together: reading the formula

Now every symbol in the parent's central formula is earned. Read it slowly:

Notice $i$ is fixed on the left (one specific slice) and $j$ sweeps everything on the bottom (the whole pizza). That is the entire job of using two different pointer letters.

6. Two more symbols the gradient section uses

The parent's gradient section introduces two notations. Build them now so that page reads cleanly.

$t_i$ in that section is the one-hot target: a probability distribution that is $1$ on the true class and $0$ everywhere else — a pizza that is entirely one slice.

Prerequisite map

Read top to bottom: lists and sums plus the exponential plus the pizza-slice idea all feed the softmax formula, which then feeds the clean gradient.

Equipment checklist

Test yourself — cover the right side.

Connections

• Parent: Softmax for output layers — the topic these foundations unlock.
• Logits and Log-Odds — where the raw $z_i$ come from.
• Maximum Entropy Distributions — the deep reason $e^z$ is the right choice.
• Cross-Entropy Loss — uses the one-hot target $t$ and the $\partial$ notation.
• Backpropagation — consumes the derivative $\partial L/\partial z_k$.
• Sigmoid Activation — the 2-class sibling built from the same $e^z$.
• Temperature Scaling — later trick that rescales the logits $z_i$.