Adversarial examples and robustness

What Are Adversarial Examples?

An adversarial example is an input $\mathbf{x}'$ created by adding a small perturbation $\boldsymbol{\delta}$ to a correctly classified input $\mathbf{x}$ , such that:

$\mathbf{x}' = \mathbf{x} + \boldsymbol{\delta}$

where $|\boldsymbol{\delta}||_p \leq \epsilon$ (the perturbation is bounded), but the model's prediction changes: $f(\mathbf{x}) \neq f(\mathbf{x}')$ .

Deriving the Fast Gradient Sign Method (FGSM)

Goal: Find the direction input space that maximally increases the loss for a given sample.

Starting from the loss function $\mathcal{L}(\theta, \mathbf{x}, y)$ where $\theta$ are model parameters,\mathbf{x} $is input,$ y$ is true label:

Step 1: Linear approximation
For small perturbations, approximate loss with first-order Taylor expansion:

$\mathcal{L}(\theta, \mathbf{x} + \boldsymbol{\delta}, y) \approx \mathcal{L}(\theta, \mathbf{x}, y) + \boldsymbol{\delta}^T \nabla_\mathbf{x} \mathcal{L}$

Why this step? We assume the loss surface is locally linear near $\mathbf{x}$ . This makes the optimization tractable.

Step 2: Maximize loss subject to $L_\infty$ constraint
We want:

$\max_{|\boldsymbol{\delta}||_\infty \leq \epsilon} \boldsymbol{\delta}^T \nabla_\mathbf{x} \mathcal{L}$

Why this step? The dot product $\boldsymbol{\delta}^T \nabla_\mathbf{x} \mathcal{L}$ is maximized when $\boldsymbol{\delta}$ points in the same direction as the gradient.

Step 3: Optimal perturbation
For $L_\infty$ ball, the optimal direction is:

$\boldsymbol{\delta} = \epsilon \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L})$

Why this step? The $\text{sign}$ function gives $+1$ or $-1$ for each component. When the gradient component is positive, we push that input dimension up by $\epsilon$ ; when negative, down by $\epsilon$ . This maximizes the dot product under $L_\infty$ constraint.

Iterative Attacks: Projected Gradient Descent (PGD)

FGSM is a single-step attack. Projected Gradient Descent (PGD) iteratively refines the perturbation for stronger attacks.

Derivation from first principles:

Initialize: $\mathbf{x}^{(0)} = \mathbf{x}$

For $t = 0, 1, \ldots, T-1$ :

Step 1: Gradient step

$\mathbf{x}^{(t+1)} = \mathbf{x}^{(t)} + \alpha \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L}(\theta, \mathbf{x}^{(t)}, y))$

Why? Take a small step $\alpha < \epsilon$ in the gradient direction.

Step 2: Project back to $\epsilon$ -ball

$\mathbf{x}^{(t+1)} = \text{Proj}_{\mathbf{x} + \mathcal{B}_\epsilon}(\mathbf{x}^{(t+1)})$

where $\mathcal{B}_\epsilon = \{\boldsymbol{\delta} : |\boldsymbol{\delta}||_\infty \leq \epsilon\}$

Why this step? The gradient might violate the perturbation budget. Projection ensures $|\mathbf{x}^{(t+1)} - \mathbf{x}||_\infty \leq \epsilon$ .

Projection formula $L_\infty$ :

$\text{Proj}_{\mathbf{x} + \mathcal{B}_\epsilon}(\mathbf{z}) = \text{clip}(\mathbf{z}, \mathbf{x} - \epsilon, \mathbf{x} + \epsilon)$

Clip each dimension: $z_i \leftarrow \max(\mathbf{x}_i - \epsilon, \min(z_i, \mathbf{x}_i + \epsilon))$

Worked example FGSM vs PGD Comparison

Setup: Image classifier, $\epsilon = 8/255$ , true class "cat" FGSM ( $\epsilon = 8/255$ , 1 step):

Compute $\nabla_\mathbf{x} \mathcal{L}$ for "cat" loss
Signs: suppose $[+1, -1, +1, -1, \ldots]$ for pixel gradients
Add $8/255 \cdot [+1, -1, +1, -1, \ldots]$ to pixels
Result: Model predicts "dog" with 60% confidence
Success rate: ~40-60% on ResNet-50

PGD ( $\epsilon = 8/255$ , $\alpha = 2/255$ , 10 steps):

Step 1: Add $2/255 \cdot \text{sign}(\nabla)$ , clip to ball
Step 2: Re-compute gradient at new point, add $2/255 \cdot \text{sign}(\nabla)$ , clip
... repeat10 times
Result: Model predicts "dog" with 95% confidence
Success rate: ~90-99% on ResNet-50

Why PGD is stronger: Multiple steps explore the loss landscape more thoroughly, finding perturbations that FGSM's single linearization mises. PGD is considered the "gold standard" for evaluating adversarial robustness.

Why Neural Networks Are Vulnerable

Three explanations (not mutually exclusive):

Linear behavior in high dimensions: Neural networks often behave linearly in local regions. The FGSM derivation assumes linearity, and it works surprisingly well, suggesting models don't have enough curvature to resist linear attacks.
Decision boundary proximity: Classification boundaries lie close to natural data manifolds in high-dimensional space. Small perturbations can cross boundaries even if images stay "natural-looking" to humans.
Insufficiently expressive features: Models may rely on non-robust features—patterns that correlate with labels in training data but aren't causally meaningful. Adversarial perturbations exploit these spurious correlations.

Defense Strategies

1. Adversarial Training

Adversarial training augments training data with adversarial examples, forcing the model to learn robust features.

Objective (Madry et al., 2018):

$\min_\theta \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ \max_{|\boldsymbol{\delta}||_\infty \leq \epsilon} \mathcal{L}(\theta, \mathbf{x} + \boldsymbol{\delta}, y) \right]$

Interpretation: Minimize the maximum loss within the $\epsilon$ -ball around each training point. The inner max finds worst-case perturbation; the outer min trains the model on it.

Training loop:

For each batch (X, y):
    1. Generate adversarial batch: X_adv = PGD(X, y, epsilon, alpha, steps)
    2. Compute loss: L = CrossEntropy(model(X_adv), y)
    3. Backprop through model (not through PGD)
    4. Update θ

Why this step? We treat PGD as a fixed data augmentation—compute adversarial examples, then do standard training on them. The inner max is not differentiated.

2. Certified Defenses

Certified defenses provide mathematical guarantees that a prediction is robust within a ball.

Randomized smoothing (Cohen et al., 2019):

Define a smoothed classifier:

$g(\mathbf{x}) = \arg\max_c \mathbb{P}(\mathbf{x} + \boldsymbol{\epsilon}) \text{ where } \boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2 I))$

Intuition: Add Gaussian noise to input, classify the noisy version, return the majority vote over many samples.

Certification theorem: If for class $c_A$ :

$\mathbb{P}(f(\mathbf{x} + \boldsymbol{\epsilon}) = c_A) \geq p_A$

and for any other class $c_B$ :

$\mathbb{P}(f(\mathbf{x} + \boldsymbol{\epsilon}) = c_B) \leq p_B$

then $g(\mathbf{x}) = c_A$ for all $\mathbf{x}'$ with $||\mathbf{x}' - \mathbf{x}||_2 \leq \frac{\sigma}{2}(\Phi^{-1}(p_A) - \Phi^{-1}(p_B))$

where $\Phi^{-1}$ is the inverse CDF of standard normal.

Why this step? The Gaussian noise "blurs" the classifier. If class $c_A$ has a strong enough probability mass, no $L_2$ perturbation smaller than the certified radius can flip the majority vote.

Trade-off: Certified radius is small (typically $< 1.0$ for ImageNet, weaker than empirical defenses) and requires expensive Monte Carlo sampling at test time.

3. Input Transformations

Examples:

JPEG compression (removes high-frequency noise)
Bit-depth reduction
Total variation denoising

Common mistake Why Simple Preprocessing Fails

Wrong idea: "If adversarial perturbations are high-frequency noise, just apply a low-pass filter!" Why it feels right: Many adversarial perturbations do have high-frequency structure, and filters do remove some perturbations.

The fix: This is an adaptive attack failure. If the attacker knows about your preprocessing, they can craft perturbations that survive it:

Add the preprocessing $T$ to the attack: $\mathbf{x}_{\text{adv}} = \mathbf{x} + \epsilon \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L}(\theta, T(\mathbf{x}), y))$
Optimize $\boldsymbol{\delta}$ such that $T(\mathbf{x} + \boldsymbol{\delta})$ fools the model

Lesson: Defenses must be evaluated against attackers who know the defense. "Security through obscurity" does not work for adversarial robustness.

Robustness Metrics

Clean accuracy: Performance on unperturbed test data
Robust accuracy: Accuracy on adversarially perturbed test data (usually PGD)
AutoAttack success rate: Standardized ensemble of attacks (Croce & Hein, 2020)

Evaluation protocol:

For each test sample, run PGD with random restarts (e.g., 10 restarts)
Report accuracy on worst-case perturbations found
Use strong attacks: PGD with many steps (50-100), multiple norms ( $L_\infty$ , $L_2$ )

Connections to AI Safety

Why adversarial robustness matters:

Specification gaming: If a model is rewarded for "high confidence" without robustness checks, it can achieve high validation accuracy while being trivially exploitable.
Out-of-distribution detection: Adversarial examples are OD inputs that models confidently misclassify. Robustness and OD detection are dual problems.
Trojan attacks: Adversarial training provides some defense against backdoor triggers (patterns that flip model behavior).
Interpretability: Understanding why adversarial examples exist helps us understand model representations. Models that rely on non-robust features aren't learning the "right" concepts.

Recall Explain to a 12-year-old

Imagine you trained a robot to recognize cats and dogs. The robot gets really good—99% accuracy! But then someone discovers a trick: if you put a tiny sticker on a dog's ear (so small you wouldn't even notice), the robot suddenly screams "CAT!" with total confidence.

That's an adversarial example. The sticker is placed so carefully that it pushes the robot's "brain" over the edge from "definitely dog" to "definitely cat," even though to your eyes, it's obviously still a dog.

The scary part? This isn't a bug—it happens in almost every AI system. The robot's brain (neural network) divides the world into regions: "this is the cat region," "this is the dog region." But these regions have super weird, jagged borders in high-dimensional space (imagine a space with 150,000 dimensions, one for every pixel!). Tiny changes that look like nothing to us can hop across those borders.

Making the robot robust means training it to recognize cats and dogs even when someone tries to trick it with stickers. It's like teaching the robot: "Don't just memorize patterns—understand what makes a dog actually a dog." This is really hard, and it's one of the big unsolved problems in AI safety.

Connections:

Interpretability and explainability — adversarial examples reveal what features models rely on
AI Safety fundamentals — robustness is a safety requirement for deployed systems
Out-of-distribution detection — adversarial examples are a form of OOD input
Reward hacking — adversarial examples are specification gaming at test time
Neural network training — adversarial training is a form of data augmentation
Gradient descent optimization — PGD is gradient ascent on loss w.r.t. inputs

#flashcards/ai-ml

What is an adversarial example? :: An input $\mathbf{x}' = \mathbf{x} + \boldsymbol{\delta}$ where the perturbation $\boldsymbol{\delta}$ is small (bounded by $\epsilon$ in some norm) but causes the model to change its prediction: $f(\mathbf{x}) \neq f(\mathbf{x}')$ . The perturbation is often imperceptible to humans.

Derive the FGSM perturbation formula from first principles :: Start with loss $\mathcal{L}(\theta, \mathbf{x}, y)$ . Linearize: $\mathcal{L}(\mathbf{x} + \boldsymbol{\delta}) \approx \mathcal{L}(\mathbf{x}) + \boldsymbol{\delta}^T \nabla_\mathbf{x} \mathcal{L}$ . To maximize this subject to $||\boldsymbol{\delta}||_\infty \leq \epsilon$ , set $\boldsymbol{\delta} = \epsilon \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L})$ . This aligns the perturbation with the gradient's direction, maximizing the dot product.

Why is PGD stronger than FGSM?

PGD takes multiple small steps (

\alpha < \epsilon

) and projects back to the

\epsilon

-ball at each iteration, exploring the loss surface more thoroughly. FGSM uses a single-step linearization that may not find the worst-case perturbation if the loss surface has curvature or if a larger total perturbation budget allows better exploration.

What is the adversarial training objective?

\min_\theta \mathbb{E}_{(\mathbf{x}, y)} \left[ \max_{||\boldsymbol{\delta}|| \leq \epsilon} \mathcal{L}(\theta, \mathbf{x} + \boldsymbol{\delta}, y) \right]

. The inner max finds the worst-case adversarial perturbation for each sample; the outer min trains the model to be robust to these perturbations.

What trade-off does adversarial training introduce?

Adversarial training typically reduces clean accuracy (by ~5-10% on CIFAR-10) while significantly improving robust accuracy under attack. It learns more conservative decision boundaries that resist perturbations but may generalize worse to unperturbed data.

What is an adaptive attack and why does it break input preprocessing defenses?

An adaptive attack is one where the attacker knows the defense mechanism (e.g., JPEG compression, denoising) and optimizes the perturbation to remain adversarial after the defense is applied. If the attacker includes the preprocessing in the attack computation (e.g.,

\nabla_\mathbf{x} \mathcal{L}(\theta, T(\mathbf{x}), y)

), simple preprocessing no longer removes the perturbation.

What does randomized smoothing provide?

Randomized smoothing provides a certified robustness guarantee. By classifying

\mathbf{x} + \boldsymbol{\epsilon}

where

\boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2 I)

and returning the majority vote, it guarantees that predictions are constant within a provable

L_2

radius. The certified radius depends on the probability gap between the top two classes.

Why are neural networks vulnerable to adversarial examples in high dimensions?

In high dimensions, even imperceptible per-dimension perturbations accumulate. The perturbation's contribution to loss

\boldsymbol{\delta}^T \nabla_\mathbf{x} \mathcal{L} = \sum_i \delta_i \frac{\partial \mathcal{L}}{\partial x_i}

can be large if all gradient components have the same sign, scaling as

\epsilon \sqrt{d}

. Neural networks often behave linearly locally, so small perturbations have predictable, large effects on loss.

What is the difference between untargeted and targeted adversarial attacks?

Untargeted attacks maximize loss to cause any misclassification:

\mathbf{x}_{\text{adv}} = \mathbf{x} + \epsilon \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L}(\theta, \mathbf{x}, y))

. Targeted attacks minimize loss for a specific target class

t

to force that prediction:

\mathbf{x}_{\text{adv}} = \mathbf{x} - \epsilon \cdot \text{sign}(\nabla_\mathbf{x} \mathcal{L}(\theta, \mathbf{x}, t))

What is robust accuracy and how is it measured?

Robust accuracy is the fraction of test samples correctly classified under worst-case adversarial perturbations within budget

\epsilon

\text{Robust-Acc} = \frac{1}{|\mathcal{D}|} \sum \mathbb{1}[\min_{||\boldsymbol{\delta}|| \leq \epsilon} f(\mathbf{x} + \boldsymbol{\delta}) = y]

. In practice, use strong attacks (PGD with many steps, multiple random restarts) as a lower bound.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, yaha basic idea ye hai ki neural networks jitne bhi powerful lagte hain, unmein ek chhupa hua weakness hai. Agar tum ek image mein bohot chhoti si, aankhon se dikhai na dene wali noise add kar do, toh model jo pehle 99% confidence se panda bol raha tha, wahi model ab 99% confidence se "gibbon" bol dega. Ye koi ek model ka bug nahi hai — ye neural networks ke high-dimensional space ko partition karne ke tareeke ki ek fundamental property hai. Isko hum adversarial example kehte hain, jaha input x mein ek chhota perturbation δ add karke prediction galat kara dete hain, aur δ ka size ek limit ε ke andar rakhte hain (jaise L-infinity norm se har pixel bas thoda sa badalta hai).

Ab attack banate kaise hain? Sabse simple method hai FGSM (Fast Gradient Sign Method). Idea simple hai — loss function ka gradient nikalo input ke respect mein, aur us gradient ki sign direction mein har pixel ko ε jitna push kar do. Kyun sign? Kyunki L-infinity constraint ke andar loss ko maximize karne ke liye tumhe har dimension mein gradient ke same direction mein full ε step lena padta hai. Bas ek hi gradient computation, aur attack ready — isliye ye "fast" hai. Aur agar tumhe model se koi specific galat class predict karwani ho, toh minus sign use karte ho taaki us target class ka loss minimize ho jaye. Isse aage PGD aata hai jo iteratively chhote-chhote steps leke aur bhi strong attack banata hai.

Ye cheez safety ke liye kyun important hai, samajhna zaroori hai. Socho agar ye adversarial trick kisi medical diagnosis system, self-driving car ke object detector, ya content moderation filter ko fool kar de — toh nateeja privacy leak se lekar physical nuksaan tak kuch bhi ho sakta hai. Isliye adversarial robustness ka pura field isliye bana hai taaki hum models ko in attacks ke against strong banaayein. Regional students ke liye takeaway ye hai — model ka high accuracy dikhna kaafi nahi hai; real-world deployment se pehle uski robustness bhi test karni padti hai, warna chhoti si noise pura system tod sakti hai.