Exercises — Convolutional neural networks — convolution operation, pooling

Answer: (b) max-pooling. Humne kya kiya: yaad kiya ki pooling ek fixed function hai — yeh ek window ke upar max (ya average) compute karta hai aur iske andar train karne ke liye kuch bhi nahi hota. Ek convolution mein har filter ke liye $k^{2}C+1$ trainable numbers hote hain; ek fully-connected layer mein har input–output pair ke liye ek weight hota hai; bias term khud ek parameter hai. Sirf pooling kuch nahi seekhta.

$W_{out}=\lfloor \frac{W_{in}-k+2p}{s}\rfloor +1$ $W_{in}$ = input width, $k$ = kernel size, $p$ = padding, $s$ = stride. Mnemonic (parent note): "I-Kicked-2-Pads-per-Stride-plus-one."

$W_{out}=\lfloor \frac{7-3+0}{1}\rfloor +1=\lfloor 4\rfloor +1=5$ Yeh kaisa dikhta hai: ek $3$-wide kernel columns $0,1,2,3,4$ pe start ho sakta hai — yeh 5 positions hain, toh 5 outputs. $-k$ woh jagah kaat deta hai jo kernel body ko chahiye; $+1$ sabse pehli position (column 0) wapas add kar deta hai, jise subtraction ne kabhi count nahi kiya tha.

$W_{out}=\lfloor \frac{32-5+2(2)}{1}\rfloor +1=\lfloor \frac{31}{1}\rfloor +1=32$ Yeh padding kyun: $p=2$ ke saath humne do zero-rings add kiye, jo bilkul utne hain ki ek $5\times 5$ kernel ke liye stride 1 par $-k$ shrink cancel ho jaye ($2p=k-1$). Yeh "same" padding hai — output size input size ke barabar hoti hai. Useful tab hota hai jab tum bahut saari conv layers stack karna chahte ho bina map ke vanish hue.

Output size: $\lfloor (3-2)/1\rfloor +1=2$, toh ek $2\times 2$ map. $K$ ko har $2\times 2$ window par overlay karo aur element-wise products ka sum karo (cross-correlation, parent §1).

• $S(0,0)=2(1)+1(-1)+1(-1)+3(1)=2-1-1+3=3$
• $S(0,1)=1(1)+0(-1)+3(-1)+1(1)=1-0-3+1=-1$
• $S(1,0)=1(1)+3(-1)+0(-1)+2(1)=1-3-0+2=0$
• $S(1,1)=3(1)+1(-1)+2(-1)+4(1)=3-1-2+4=4$ $S=\left[\begin{array}{cc}3& -1\\ 0& 4\end{array}\right]$ Yeh kaisa dikhta hai: yeh kernel positively fire karta hai jab do diagonal pixels bade hain aur do anti-diagonal pixels chote hain — ek checkerboard / corner detector.

Stride 2 ke saath $2\times 2$ window ⇒ non-overlapping $2\times 2$ blocks. Output $\lfloor (4-2)/2\rfloor +1=2$, ek $2\times 2$ map. Max-pool (har block mein sabse bada rakho):

• TL $\{1,3,4,2\}\to 4$, TR $\{2,1,0,5\}\to 5$
• BL $\{6,1,0,7\}\to 7$, BR $\{3,2,1,1\}\to 3$ $P_{max}=\left[\begin{array}{cc}4& 5\\ 7& 3\end{array}\right]$ Average-pool (har block ka mean):
• TL $(1+3+4+2)/4=2.5$, TR $(2+1+0+5)/4=2$
• BL $(6+1+0+7)/4=3.5$, BR $(3+2+1+1)/4=1.75$ $P_{avg}=\left[\begin{array}{cc}2.5& 2\\ 3.5& 1.75\end{array}\right]$

$\#\text{params}=F(k^{2}C+1)=64(3^2\cdot 3+1)=64(27+1)=64\cdot 28=1792$ $512\times 512$ image ke liye: abhi bhi 1792. Formula $F(k^{2}C+1)$ mein koi $W_{in}$ nahi hota. Kyun: wahi kernel poori image par slide kiya jaata hai (weight sharing), toh image bada karne ka matlab hai zyada slides, nahi zyada weights. Yeh size-independence convolution ka fully-connected layer ke upar defining advantage hai.

$\max (S)=8$ aur $\max (S^{\prime })=8$. Dono pool hokar same value 8 dete hain. Yeh kya dikhata hai: local translation invariance — feature (strong activation "8") ek pixel move hua, phir bhi pooled output nahi badla. Max-pooling report karta hai "yeh feature is window mein present tha" bina is baat ki parwah kiye ki window mein exactly kahan tha. Figure dekho.

Ek $3\times 3$ conv ek neuron ko width 3 ki receptive field deta hai. Use second $3\times 3$ conv mein feed karo: second neuron 3 first-layer neurons combine karta hai, aur woh 3 neighbours input columns $[0,2],[1,3],[2,4]$ cover karte hain — milke columns $0$ se $4$ tak, yani width 5. Stacked stride-1 conv ka general rule: receptive field har layer mein $(k-1)$ se badhta hai, toh do $3\times 3$ layers ke baad yeh $3+(3-1)=5$ hoti hai. Do chhote kernels utna door dekhte hain jitna ek $5\times 5$ kernel, lekin kam parameters mein — isliye deep stacks of small kernels prefer ki jaati hain.

$(W\times H\times \text{channels})$ track karo. Start: $64\times 64\times 1$.

• Stage 1 conv ($k=3,p=1,s=1$ = "same"): $\lfloor (64-3+2)/1\rfloor +1=64$. → $64\times 64\times 16$.
• Stage 2 pool ($2\times 2,s=2$): $\lfloor (64-2)/2\rfloor +1=32$. → $32\times 32\times 16$.
• Stage 3 conv (same padding): $32\times 32\times 32$.
• Stage 4 pool: $\lfloor (32-2)/2\rfloor +1=16$. → $16\times 16\times 32$.

Parameters (pooling ke koi nahi hote):

• Conv 1: $16(3^2\cdot 1+1)=16\cdot 10=160$
• Conv 3: $32(3^2\cdot 16+1)=32\cdot 145=4640$
• Total $=160+4640=4800$.

Yahan "same"-padding kyun matter karta hai: yeh conv layers ko spatial size fixed rakhne deta hai, toh saari shrinking pooling ke zariye ek controlled, predictable $÷2$ pattern mein hoti hai — reason karna aasan ho jaata hai.

Input mein $64\times 64\times 1=4096$ units hain. Output mein $64\times 64\times 16=65536$ units hain. Fully-connected weights $=4096\times 65536=268,435,456\approx 2.68\times 10^8$. Conv ne 160 use kiye. Ratio $\approx 1.68\times 10^6$ — conv ek million times se bhi zyada lean hai. Kyun: locality (har output sirf ek $3\times 3$ neighbourhood touch karta hai) + weight sharing (ek kernel har jagah reuse hota hai) milke us quarter-billion matrix ko 160 numbers mein compress kar dete hain. Yeh parent ka "convolution is a constrained fully-connected layer" concrete form mein hai.

Stride $s=2$ try karo (roughly half karne ke liye). $p$ ke liye solve karo jo exactly 112 deta hai: $112=\lfloor \frac{224-3+2p}{2}\rfloor +1\Rightarrow \lfloor \frac{221+2p}{2}\rfloor =111$ Humein chahiye $\frac{221+2p}{2}\in [111,112)$ ⇒ $221+2p\in [222,224)$ ⇒ $2p\in [1,3)$ ⇒ $p=1$. $p=1,s=2$ se check karo: $\lfloor (224-3+2)/2\rfloor +1=\lfloor 223/2\rfloor +1=\lfloor 111.5\rfloor +1=111+1=112.$ ✓ Answer: $p=1,s=2$. Stride 2 kyun, padding tricks nahi: halving fundamentally ek stride ka kaam hai (stride rate of down-sampling set karta hai); padding phir edge fine-tune karta hai taaki floor exactly 112 par land kare.

Max-pooling choose karo. Ek crack ek sparse, high-contrast feature hai: panel ka ek patla bright line covering ek chhota sa fraction. Us line ke liye tuned ek conv filter ek ya kuch bade activations produce karta hai jo near-zero background se ghiri hoti hain.

• Max-pool us bade activation ko intact rakhta hai — yeh jawaab deta hai "kya crack feature is window mein kabhi aaya?", jo bilkul wahi yes/no sawaal hai.
• Average-pool strong activation ko poore window ki cell count se divide kar deta hai, ek akele hot pixel ko zeros ke samandar mein duba deta hai. Ek real crack averaged hokar decision threshold se neeche aa sakti hai — ek missed defect, inspection mein sabse bura error. Jab average jeet jaata hai: overall magnitude / texture ke baare mein tasks (jaise mean surface roughness estimate karna), jahan smoothing ek bug nahi, balki ek feature hai.

Conv2d cross-correlation compute karta hai (kernel flip nahi hota), jabki textbook ki true convolution pehle kernel ko 180° flip karti hai (parent §1). Ek hand-designed asymmetric kernel jaise Sobel ke liye, cross-correlation mirrored filter apply karta hai — detected edge orientation invert ho sakti hai. Hand-coded kernels ke liye fix: kernel ko (180° rotate karke) load karne se pehle pre-flip karo. Ek learned network ke liye: yeh bilkul matter nahi karta — gradient descent simply woh weights ka orientation seekhta hai jo loss minimize kare, yani yeh automatically flipped values seekh leta hai. Flip sirf tab concern hai jab tum ek fixed, meaningful kernel inject karte ho.

Answers cover karo: $W_{out}=?$ ::: $\lfloor (W_{in}-k+2p)/s\rfloor +1$ Ek conv layer ke params ::: $F(k^{2}C+1)$, image-size independent Pooling params ::: zero Sparse crack signal ke liye best pool ::: max