Adversarial examples and robustness: Question bank

Before the questions, one shared picture so the words below have something to point at.

Imagine every possible image is a single dot in a vast space. A decision boundary is the wall that separates "cat dots" from "dog dots". A perturbation $δ$ is a tiny arrow you add to a dot to nudge it. An == $ϵ$ -ball== is a small box drawn around the original dot: you are only allowed to move inside that box. An attack succeeds when a tiny arrow, still inside the box, pushes the dot across the wall.

Keep three symbols in mind, all defined in the parent note:

$\nabla_{x} L$ — the gradient of the loss with respect to the input: an arrow pointing in the direction that makes the model more wrong.
$sign (\cdot)$ — replaces every number by just $+ 1$ , $- 1$ , or $0$ (its sign).
$ϵ$ (epsilon) — the budget: how far you may move along any single pixel.

True or false — justify

The model needs access to a specific "hard" image to build an adversarial example

False — any correctly classified input can be perturbed; the gradient direction is computed from that input, so every clean image has its own custom attack.

L_{\infty}

bound of

ϵ = 8/255

means the total change across all pixels is

8/255

False —

L_{\infty}

bounds the change to each pixel independently (

max_{i} ∣ δ_{i} ∣

), so every one of the 150{,}528 numbers may move by the full

8/255

at once.

If a perturbation is invisible to a human, the model must also see it as basically unchanged

False — human vision and the network read different features; the whole phenomenon is that a change below human perception can flip the network entirely.

Making

ϵ

smaller always makes the attack fail

False in spirit — smaller

ϵ

makes attacks harder, but even minuscule budgets can succeed when the clean point already sits very close to a decision boundary.

FGSM guarantees the model is fooled after one step

False — one linear step often works but has only ~40–60% success on ResNet-50; the linear approximation can point in a direction that stops helping before the boundary is crossed.

PGD can escape the

ϵ

-ball during its search

False — after every gradient step PGD projects (clips) back into the ball, so the final perturbation always satisfies

∥ δ ∥_{\infty} \leq ϵ

Adversarial training makes clean accuracy go up

False — it typically lowers clean accuracy (e.g. 95% → 87% on CIFAR-10) because robust features are more conservative; robustness and clean accuracy trade off.

A model with 0% accuracy under PGD is simply a badly trained model

False — even a 95%-clean, well-trained standard model can hit 0% under PGD; vulnerability is a property of the geometry, not of poor optimization.

The curse of dimensionality helps robustness because errors average out

False — it works against robustness: per-pixel nudges of size

ϵ / d

can sum to

ϵ d

when gradient signs align, which grows with dimension.

Targeted and untargeted FGSM use the same sign in the update

False — untargeted adds

ϵ sign (\nabla)

to raise the true-class loss, targeted subtracts it to lower the loss of the desired target class.

Spot the error

"PGD uses step size

α = 8/255

equal to

ϵ

so it moves fast." — what's wrong?

You want

α < ϵ

so multiple steps can refine within the ball;

α = ϵ

collapses PGD into a jerky single-step attack and wastes the projection stage.

"Adversarial training differentiates through the PGD attack to update

θ

." — what's wrong?

The inner max (PGD) is treated as fixed data augmentation; you backprop through the model on the generated

X_{adv}

, not through the PGD steps themselves.

"We add

ϵ \cdot \nabla_{x} L

in FGSM." — what's wrong?

FGSM uses

ϵ \cdot sign (\nabla_{x} L)

; using the raw gradient lets large components blow past the

L_{\infty}

budget, so the sign is essential to respect the per-pixel bound.

"The

L_{\infty}

optimal perturbation points exactly along the gradient vector." — what's wrong?

Under an

L_{\infty}

box the optimum is at a corner of the box, i.e.

\pm ϵ

on each axis — that is the sign vector, which generally differs from the gradient's direction.

"Adversarial examples exist only for images." — what's wrong?

The math needs only a differentiable loss and continuous inputs; audio, text embeddings, malware features and tabular models are all attackable.

"Projection in

L_{\infty}

means rescaling

δ

to length

ϵ

." — what's wrong?

Rescaling is the

L_{2}

picture;

L_{\infty}

projection is coordinate-wise clipping into

[x_{i} - ϵ, x_{i} + ϵ]

, not a global rescale.

"A model robust at

ϵ = 8/255

is robust at all budgets." — what's wrong?

Robustness is budget-specific; a larger

ϵ

gives a bigger box and usually breaks the same model, so claims must state their

ϵ

Why questions

Why is the dot product

δ^{T} \nabla_{x} L

the right thing to maximize?

The first-order Taylor expansion says the loss change from a small step equals this dot product, so maximizing it means making the model as wrong as possible per unit of budget.

Why does the sign function appear instead of keeping gradient magnitudes?

Under an

L_{\infty}

constraint each coordinate is spent up to

\pm ϵ

regardless of how steep it is; the optimal choice is full budget in the sign direction, so magnitude is discarded.

Why is PGD called the "gold standard" for evaluating robustness?

Its many projected steps explore the loss surface thoroughly, so a defense that survives strong PGD is unlikely to be a false sense of security from a weak attacker.

Why does high dimensionality make attacks easier, not harder?

With many coordinates, many small aligned nudges add up (

\sim ϵ d

) even though each is imperceptible, so the loss can be moved a lot within a tiny per-pixel box.

Why does adversarial training use the inner max rather than random noise?

Random noise trains against the average case; the inner max trains against the worst case in the

ϵ

-ball, which is exactly what a real attacker searches for.

Why can a model that "looks" interpretable still be adversarially fragile?

It may rely on non-robust features — patterns that correlate with labels but aren't causal — and perturbations exploit precisely those spurious correlations (see Interpretability and explainability).

Why does clean accuracy usually drop after adversarial training?

Fitting worst-case points forces smoother, more conservative decision boundaries, which sacrifices some fine distinctions that boosted clean accuracy.

Edge cases

What is

sign (0)

and why does it matter for FGSM?

It is

0

, so a pixel with exactly zero gradient gets no perturbation — those coordinates contribute nothing to the attack and stay untouched.

What happens if the model is already wrong on the clean input?

There is nothing to "flip" — the FGSM/PGD framing assumes a correctly classified

x

; on an already-misclassified point the notion of a successful untargeted attack is undefined.

What if the loss surface is genuinely nonlinear near

x

FGSM's single linear step may point somewhere unhelpful; PGD's iterated re-computation of the gradient at each new point corrects course, which is why it succeeds where FGSM misses.

What if the gradient is (nearly) zero everywhere around

x

(gradient masking)?

Gradient-based attacks stall and the model looks robust, but this is a false defense — transfer or gradient-free attacks often still break it, so flat gradients are a warning sign, not a win.

What does

ϵ = 0

correspond to?

A zero-size box: the only allowed perturbation is

δ = 0

, so

x_{adv} = x

and no attack is possible by definition.

After projection, can a pixel value leave the valid image range (like below 0 or above 1)?

Yes — the

ϵ

-clip alone doesn't enforce it, so a second clip to the legal pixel range (e.g.

[0, 1]

) is needed to keep

x_{adv}

a real image.

If all gradient components have opposite signs to each other, does the high-dimensional

ϵ d

blow-up still happen?

No — the dramatic accumulation needs the signed contributions to align; mixed signs let terms cancel, shrinking the achievable loss increase.

Recall One-line self-test

Under $L_{\infty}$ , why is the optimal single-step perturbation $ϵ sign (\nabla_{x} L)$ and not $ϵ \nabla_{x} L$ ? ::: Because the constraint is a box, the loss-maximizing point sits at a corner (full $\pm ϵ$ per axis) = the sign vector; the raw gradient would violate the per-pixel budget.