DenseNet and EfficientNet

3.4.10 · AI-ML › Convolutional Neural Networks

Overview

Do landmark architectures jo CNN design ko revolutionize kar gaye: DenseNet dense connections ke through feature reuse maximize karta hai, jabki EfficientNet networks ko systematically scale karta hai optimal accuracy-efficiency tradeoff ke liye.

DenseNet: Dense Convolutional Networks

Dense block mein layer $\ell$ ke liye: ${\mathbf{x}}_{\ell }=H_{\ell }([{\mathbf{x}}_0,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{\ell -1}])$

jahan $[{\mathbf{x}}_0,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{\ell -1}]$ channel dimension ke saath concatenation denote karta hai, aur $H_{\ell }$ ek composite function hai: BN → ReLU → Conv3×3.

Ye kyun important hai:

• Gradient highway: Gradients directly early layers tak flow kar sakte hain (vanishing gradients kam hote hain)
• Feature reuse: Late layers early layers ke raw low-level features access kar sakti hain
• Parameter efficiency: Redundant features dobara seekhne ki zaroorat nahi

Growth Rate aur Parameters Derive Karna

First principles se derivation:

• Layer 0: input mein $k_0$ channels hain
• Layer 1: $k_0$ channels receive karta hai, $k$ naye channels output karta hai → total = $k_0+k$
• Layer 2: $k_0+k$ channels receive karta hai, $k$ naye channels output karta hai → total = $k_0+2k$
• Layer $\ell$: $k_0+(\ell -1)k$ channels receive karta hai, $k$ output karta hai → total = $k_0+\ell k$
• $L$ layers ke baad: $k_0+Lk$ channels

Chhota $k$ kyun kaam karta hai: Typical $k=12$ ya $32$ hota hai. Har layer "collective knowledge" mein feature maps ka ek chhota set contribute karti hai, lekin sabhi predecessors se rich combined feature set access karti hai.

$H_{\ell }:\text{BN-ReLU-Conv1×1}\to 4k\text{ channels}\to \text{BN-ReLU-Conv3×3}\to k\text{ channels}$

1×1 convolution kyun? Expensive 3×3 convolution se pehle dimensionality reduce karta hai.

Ek bottleneck layer ke liye parameter calculation:

• Agar input mein $m$ channels hain:
  • 1×1 conv: $m\times 4k\times 1\times 1=4mk$ parameters
  • 3×3 conv: $4k\times k\times 3\times 3=36k^2$ parameters
• Bottleneck ke bina: $m\times k\times 3=9mk$ parameters

Jaise $m$ badhta hai (block mein late stages mein), reasonable $k$ ke liye $4mk\ll 9mk$, toh bottlenecks parameters bachate hain.

Agar ek dense block $m$ channels output karta hai, toh transition layer $\lfloor \theta m\rfloor$ channels tak compress karta hai jahan $\theta \in (0,1]$ compression factor hai (typically $\theta =0.5$).

Compression kyun? Model size aur computational cost control karta hai. Har dense block ke baad channels ko half karna growth ko manageable rakhta hai.

Total layers: $1+(6+12+24+16)=59$ conv layers dense blocks ke andar, plus transitions ≈ bottlenecks ke saath 121 layers.

Ye configuration kyun kaam karti hai: Early blocks (kam layers) low/mid-level features seekhte hain. Deep blocks (zyada layers) high-level abstractions seekhte hain. Har ek ke paas sabhi prior features ka access hota hai.

Ye step kyun? Har layer apna output ($k=8$) running total mein concatenate karta hai.

$\theta =0.5$ ke saath transition ke baad: $48\times 0.5=24$ channels.

Ye sahi kyun lagta hai: Dono "skip" connections hain jo gradients mein help karte hain.

Sahi baat: DenseNet feature maps ko concatenate karta hai: ${\mathbf{x}}_{\ell }=H_{\ell }([{\mathbf{x}}_0,\dots ,{\mathbf{x}}_{\ell -1}])$. Ye sabhi prior information explicitly preserve karta hai, jabki ResNet ka addition features ko merge karta hai (individual identity kho jaati hai). Concatenation true feature reuse enable karta hai lekin memory zyada lagti hai; addition memory-efficient hai lekin kam reuse hota hai.

EfficientNet: Compound Scaling

$\text{depth: }d={\alpha }^{\varphi }$ $\text{width: }w={\beta }^{\varphi }$ $\text{resolution: }r={\gamma }^{\varphi }$

constraint ke saath: $\alpha \cdot {\beta }^2\cdot {\gamma }^2\approx 2$, jahan $\alpha \ge 1,\beta \ge 1,\gamma \ge 1$.

Ye constraint kyun? $\varphi$ mein har unit increase ke saath FLOPs doubling approximate karta hai.

Scaling Law Derive Karna

Ek layer ke liye total FLOPs: $\text{FLOPs}\propto d\cdot w^2\cdot r^2$

Agar hum har dimension scale karte hain: ${\text{FLOPs}}_{\text{new}}\propto (d\cdot {\alpha }^{\varphi })\cdot (w\cdot {\beta }^{\varphi }{)}^2\cdot (r\cdot {\gamma }^{\varphi }{)}^2$ $=d\cdot w^2\cdot r^2\cdot {\alpha }^{\varphi }\cdot {\beta }^{2\varphi }\cdot {\gamma }^{2\varphi }$ $={\text{FLOPs}}_{\text{base}}\cdot (\alpha \cdot {\beta }^2\cdot {\gamma }^2{)}^{\varphi }$

$\varphi$ ke har increment par FLOPs double karne ke liye: $\alpha \cdot {\beta }^2\cdot {\gamma }^2=2$

Ye kyun important hai: Ek given computational budget (FLOPs) ke liye, ye formula batata hai ki maximum accuracy ke liye depth, width, aur resolution mein scaling kaise distribute karein.

Check karo: $1.2\times 1.1^2\times 1.15^2=1.2\times 1.21\times 1.3225\approx 1.92\approx 2$ ✓

$\varphi =1$ ke saath EfficientNet-B1 create karne ke liye:

• Depth: $d_{B1}=1.2^1\times d_{B0}=1.2\times d_{B0}$
• Width: $w_{B1}=1.1^1\times w_{B0}=1.1\times w_{B0}$
• Resolution: $r_{B1}=1.15^1\times 224\approx 240$ pixels

$\varphi =6$ ke saath EfficientNet-B7 ke liye:

• Depth: $1.2^6\approx 3.0\times d_{B0}$
• Width: $1.1^6\approx 1.77\times w_{B0}$
• Resolution: $1.15^6\times 224\approx 456$ pixels

Ye step kyun? Har dimension $\varphi$ ke saath exponentially badhta hai, lekin constrained hai taki total FLOPs bhi exponentially badhe ($\varphi$ ke har increment mein ≈ 2× FLOPs).

1. Expand: 1×1 conv se channels ko factor $t$ se expand karo (typically 6)
2. Depthwise: 3×3 ya 5×5 depthwise conv (ek filter per channel)
3. Squeeze-and-Excitation (SE): channel attention
4. Project: 1×1 conv se output channels par project wapas karo
5. Skip connection agar input/output dimensions match karein

MBConv kyun? Extremely parameter-efficient. Depthwise separable convolutions standard convolutions ke comparison mein FLOPs dramatically reduce karte hain.

Strategy 1: Sirf Depth ($\alpha =2,\beta =1,\gamma =1$)

• FLOPs: $2\times 1^2\times 1^2=2$ ✓
• Result: Bahut deep, narrow network. Risk: vanishing gradients, train karna mushkil.

Strategy 2: Sirf Width ($\alpha =1,\beta =\sqrt{2},\gamma =1$)

• FLOPs: $1\times 2\times 1=2$ ✓
• Result: Wide, shallow network. Risk: limited representational depth.

Strategy 3: Sirf Resolution ($\alpha =1,\beta =1,\gamma =\sqrt{2}$)

• FLOPs: $1\times 1\times 2=2$ ✓
• Result: Higher resolution, same depth/width. Risk: fine spatial details hain lekin koi added abstraction layers nahi.

Strategy 4: Compound scaling ($\alpha =1.2,\beta =1.1,\gamma =1.15$, $\varphi =1$)

• FLOPs: $1.2\times 1.21\times 1.3225\approx 2$ ✓
• Result: Sabhi dimensions mein balanced growth. Empirically higher accuracy achieve karta hai.

Compound scaling kyun jeetta hai: Depth abstraction add karta hai, width har layer mein capacity add karti hai, resolution finer patterns capture karta hai. Teeno scale karne se kisi bhi single dimension mein bottleneck avoid hota hai.

Ye sahi kyun lagta hai: Zyada better hota hai, hai na?

Sahi baat: Unconstrained scaling FLOPs explode kar deta hai: $2\times 4\times 4=32\times$ FLOPs increase (2× nahi). Tum apna compute budget uda dete ho. $\alpha {\beta }^2{\gamma }^2\approx 2$ constraint ensure karta hai ki tum ek target computational envelope ke andar scale karo. Empirically, exponents 1, 2, 2 reflect karte hain ki depth, width, aur resolution convolutional layers mein FLOPs ko kaise impact karte hain.

DenseNet aur EfficientNet Compare Karna

| Aspect | DenseNet | EfficientNet | |--------|--------------| | Core Idea | Feature reuse ke liye dense connectivity | Efficiency ke liye compound scaling | | Parameter Efficiency | Feature reuse se kam params | MBConv + AutoML se kam params | | Memory | High (concatenation sabhi intermediate features store karta hai) | Low (inverted bottlenecks) | | Training | Memory-intensive ho sakta hai | Efficient, lekin careful scaling chahiye | | Strength | Excellent gradient flow, feature reuse | State-of-the-art accuracy/FLOP tradeoff | | Use Case | Jahan feature reuse matter karta hai (medical imaging, fine-grained classification) | Resource-constrained ya production deployment |

DenseNet: Har baar jab tum LEGO bricks ki ek nayi layer add karte ho, tum usse sirf neeche wali layer se nahi jodte, balki tum strings (connections) neeche ki har layer tak chalate ho. Toh top brick bottom se, middle se, sab se connected hai. Matlab har brick dekh sakti hai ki sabhi earlier bricks ne kya seekha. Ye bilkul aise hai jaise class mein sablog apne notes sabke saath share karein—koi bhi important info miss nahi karta. Downside? Tumhe bahut zyada string (memory) chahiye.

EfficientNet: Ab socho tum apna tower aur lamba banana chahte ho. Tum bas aur layers stack kar sakte ho (use deeper bana sakte ho). Ya har layer ko wider bana sakte ho (zyada bricks per layer). Ya bade bricks use kar sakte ho (higher resolution images). EfficientNet kehta hai: teeno karo, lekin balanced tarike se. Agar tum usse 20% deeper banate ho, toh usse 10% wider banao aur 15% bade bricks use karo, sab ek saath. Is tarah tower mazboot aur balanced rehta hai bina girne ke ya zyada material use kiye.

Dono smart hone ke baare mein hain: DenseNet sab kuch share karta hai, EfficientNet sab kuch proportionally scale karta hai.

Connections

• ResNet - DenseNet skip connections ko full dense connectivity tak extend karta hai
• Inception - Multi-scale processing, dono 1×1 bottlenecks use karte hain
• MobileNet - EfficientNet MobileNetV2 ke MBConv blocks par build karta hai
• Neural Architecture Search - EfficientNet baseline architecture find karne ke liye NAS use karta hai
• Batch Normalization - Dono architectures mein critical component
• Model Compression - Dono architectures efficiency research ko motivate karte hain

#flashcards/ai-ml

DenseNet mein dense connectivity pattern kya hai? :: Har layer channel dimension ke saath concatenation ke zariye sabhi preceding layers se feature maps receive karti hai, aur apna output sabhi subsequent layers ko pass karti hai.

EfficientNet mein compound scaling kya hai? :: Network depth ($d={\alpha }^{\varphi }$), width ($w={\beta }^{\varphi }$), aur resolution ($r={\gamma }^{\varphi }$) ko ek compound coefficient $\varphi$ use karke ek saath uniformly scale karna, $\alpha {\beta }^2{\gamma }^2\approx 2$ ke constraint ke saath FLOPs control karne ke liye.

MBConv blocks kya hain aur EfficientNet inhe kyun use karta hai? :: Mobile Inverted Bottleneck Convolution blocks: channels expand karo (1×1 conv), depthwise conv apply karo (3×3 ya 5×5), SE attention add karo, neeche project karo (1×1 conv), skip connection. Depthwise separable convolutions ki wajah se extremely parameter-efficient hote hain.

Concept Map