Overview

TensorFlow is Google's open-source numerical computation library optimized for machine learning, while Keras is a high-level neural network API that runs on top of TensorFlow (and other backends). As of TensorFlow 2.x, tf.keras is the official high-level API integrated directly into TensorFlow.

WHY this matters: ~95% of production deep learning uses frameworks. Writing backprop by hand is educational but impractical. TensorFlow handles the computational graph, device placement, and optimization so you focus on architecture.

Core Concepts

1. Tensors: The Fundamental Data Structure

Shape hierarchy:

• Rank 0: scalar (single number)
• Rank 1: vector (1D array)
• Rank 2: matrix (2D array)
• Rank 3+: higher-order tensors (images, videos, batches)

WHY immutable? Allows TensorFlow to optimize the computation graph—once defined, operations can be compiled and paralelized without worying about side effects.

import tensorflow as tf
 
# Creating tensors
scalar = tf.constant(42)                # Shape: ()
vector = tf.constant([1, 2, 3])            # Shape: (3,)
matrix = tf.constant([[1, 2], [3, 4]])     # Shape: (2, 2)
batch_images = tf.zeros((32, 28, 28, 1))   # Shape: (batch, height, width, channels)

Element-wise operations

c = a + b # 6.0, 8.0, [10.0, 12.0]] d = a * b # Element-wise multiplication

Matrix operations

e = tf.matmul(a, b) # Matrix multiplication f = tf.transpose(a) # Transpose

**Why this step?** Each operation creates a new tensor node in the computation graph. TensorFlow tracks dependencies for automatic differentiation.

### 2. Building Models: Three Approaches

#### Sequential API (Simplest)

> [!definition] Sequential Model
> A ==Sequential model== is a linear stack of layers where each layer has exactly one input tensor and one output tensor.

**DERIVATION from first principles:**
A neural network is a composition of functions: $y = f_n(f_{n-1}(...f_2(f_1(x))))$

In code, this becomes:
```python
from tensorflow import keras

model = keras.Sequential([
    keras.layers.Dense(128, activation='relu', input_shape=(784,)),  # f₁
    keras.layers.Dropout(0.2),                                # regularization
    keras.layers.Dense(64, activation='relu'),                       # f₂
    keras.layers.Dense(10, activation='softmax')                     # f₃ (output)
])

WHY this structure?

• input_shape=(784,): First layer needs to know input dimensions (flattened 28×28 image)
• Dense(128): Fully connected layer with 128 neurons → learns784×128 + 128 = 100,480 parameters
• activation='relu': Non-linearity (without it, stacking layers = single linear transformation)
• Dropout(0.2): Randomly zeros20% of activations during training → prevents overfitting
• softmax: Converts logits to probability distribution over 10 classes

$\mathbf{z} = \mathbf{x} \mathbf{W} + \mathbf{b}$ $\mathbf{a} = \sigma(\mathbf{z})$

where $\mathbf{x}$ is the input (batch_size × n), $\sigma$ is the activation function.

Derivation: Each neuron $j$ computes $z_j = \sum_{i=1}^{n} w_{ij} x_i + b_j$, vectorized as matrix multiplication for efficiency.

Functional API (Flexible)

inputs = keras.Input(shape=(784,))
x = keras.layers.Dense(128, activation='relu')(inputs)
x = keras.layers.Dropout(0.2)(x)
x = keras.layers.Dense(64, activation='relu')(x)
outputs = keras.layers.Dense(10, activation='softmax')(x)
 
model = keras.Model(inputs=inputs, outputs=outputs)

WHY use this? Enables complex architectures:

# Multi-input example
input1 = keras.Input(shape=(128,))
input2 = keras.Input(shape=(64,))
x1 = keras.layers.Dense(64)(input1)
x2 = keras.layers.Dense(64)(input2)
merged = keras.layers.concatenate([x1, x2])
output = keras.layers.Dense(1)(merged)
 
model = keras.Model(inputs=[input1, input2], outputs=output)

Model Subclassing (Most Control)

class CustomModel(keras.Model):
    def __init__(self):
        super().__init__()
        self.dense1 = keras.layers.Dense(128, activation='relu')
        self.dropout = keras.layers.Dropout(0.2)
        self.dense2 = keras.layers.Dense(10, activation='softmax')
    def call(self, inputs, training=False):
        x = self.dense1(inputs)
        if training:  # Only apply dropout during training
            x = self.dropout(x, training=training)
        return self.dense2(x)

WHY subclass? Dynamic computation graphs, custom training loops, research experimentation.

3. Training Pipeline: Compile → Fit → Evaluate

1. Loss function $\mathcal{L}(\theta)$: Measures prediction error
2. Optimizer: Updates parameters $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$
3. Metrics: Monitors performance (accuracy, F1, etc.)

DERIVATION of gradient descent: Starting from Taylor expansion: $\mathcal{L}(\theta - \eta \nabla\mathcal{L}) \approx \mathcal{L}(\theta) - \eta |\nabla\mathcal{L}||^2$

For small $\eta > 0$, moving opposite to gradient decreases loss.

# Compile: Define learning algorithm
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=['accuracy']
)

WHY these choices?

• Adam optimizer: Adaptive learning rates per parameter (combines momentum + RMSprop)
• SparseCategoricalCrossentropy: For integer labels (0-9) instead of one-hot vectors
• Accuracy metric: % of correct predictions (more interpretable than loss)

Load data

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

Preprocessing

x_train = x_train.reshape(-1, 784).astype('float32') / 255.0 # Flatten & normalize x_test = x_test.reshape(-1, 784).astype('float32') / 255.0

Build model

model = keras.Sequential([ keras.layers.Dense(128, activation='relu', input_shape=(784,)), keras.layers.Dropout(0.2), keras.layers.Dense(10, activation='softmax') ])

Compile

model.compile( optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'] )

Train

history = model.fit( x_train, y_train, batch_size=32, epochs=5, validation_split=0.2, # Use 20% of training data for validation verbose=1 )

Evaluate

test_loss, test_acc = model.evaluate(x_test, y_test) print(f"Test accuracy: {test_acc:.4f}")

**Step-by-step WHY:**
1. **Reshape to (6000, 784):** Dense layers expect flat vectors, not 2D images
2. **Divide by 255:** Normalizes pixels to [0,1] range → faster convergence (gradients better scaled)
3. **batch_size=32:** Process 32 examples at a time → balance between memory and gradient noise
4. **epochs=5:** One epoch = one pass through entire training set
5. **validation_split=0.2:** Monitor overfitting by tracking loss on unseen data

### 4. Key Operations

> [!formula] Common Layer Types

**Dense (Fully Connected):**
$$\mathbf{y} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$$
Parameters: $n_{in} \times n_{out} + n_{out}$

**Dropout:**
During training: $\mathbf{y}_i = \begin{cases} 0 & \text{with probability } p \\ \frac{\mathbf{x}_i}{1-p} & \text{otherwise} \end{cases}$

**WHY divide by $(1-p)$?** Maintains expected value: $\mathbb{E}[\mathbf{y}_i] = \mathbb{E}[\mathbf{x}_i]$ (inverted dropout)

**Softmax (output layer for classification):**
$$\text{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$

**Derivation:** We want outputs that:
1. Sum to 1 (probability distribution)
2. Are positive
3. Preserve ordering (larger logit → larger probability)

Exponential satisfies (2) and (3), normalization satisfies (1).

## Common Patterns

### Data Pipeline
```python
# Create tf.data.Dataset for efficient loading
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(1000).batch(32).prefetch(tf.data.AUTOTUNE)

# Use in training
model.fit(train_dataset, epochs=5)

WHY this approach?

• shuffle: Randomizes order → prevents model from learning sequence patterns
• batch: Groups examples → efficient GPU computation
• prefetch: Loads next batch while training current → hides I/O latency

Callbacks

callbacks = [
    keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True),
    keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True),
    keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=2)
]
 
model.fit(x_train, y_train, epochs=100, callbacks=callbacks)

WHY use callbacks?

• EarlyStopping: Stops training when validation loss stops improving (prevents overfitting)
• ModelCheckpoint: Saves best model during training (not just the last epoch)
• ReduceLROnPlateau: Decreases learning rate when stuck → helps escape plateaus

WHY it feels right: The network should learn the appropriate scale through training.

The problem: Large input values → large gradients → unstable training or exploding gradients. Optimizer learning rates are tuned for normalized inputs.

The fix:

x = x / 255.0  # Scale to [0, 1]
# OR
x = (x - mean) / std  # Standardize to zero mean, unit variance

Steel-man: If you carefully tune the learning rate and weight initialization for the specific input scale, unnormalized data can work—but it's fragile and slows convergence.

But labels are integers: [3, 7, 2 ...]

**WHY it feels right:** "Categorical" sounds right for categories.

**The problem:** `categorical_crossentropy` expects shape (batch_size, num_classes) with one-hot encoding. Integer labels have shape (batch_size,).

**The fix:**
```python
# For integer labels: use sparse version
model.compile(loss='sparse_categorical_crossentropy', ...)

# OR one-hot encode labels first
y_train_onehot = keras.utils.to_categorical(y_train, num_classes=10)
model.compile(loss='categorical_crossentropy', ...)

The problem: Dropout randomly zeros neurons → stochastic predictions. At test time, we want deterministic, averaged predictions.

What Keras does automatically:

• Training: Dropout active (with scaling)
• Inference (model.predict(), model.evaluate()): Dropout disabled

Manual control:

# Force training mode
predictions = model(x, training=True)  # Dropout active
 
# Force inference mode
predictions = model(x, training=False)  # Dropout disabled

That's where Keras comes in—it's like a simple instruction manual. Instead of saying "take this28×28 grid of pixels, flatten it into784 numbers, multiply by this matrix, add these numbers, apply this squishing function..." you just say Dense(128, activation='relu') and Keras handles all the details.

Training a model is like teaching:

1. Show the robot lots of examples ("this is a 3", "this is a 7")
2. It makes guesses and you tell it when it's wrong
3. It adjusts its "brain connections" (weights) to do better next time
4. After seeing thousands of examples, it gets really good at recognizing new numbers it's never seen

The batch_size is like studying in groups—instead of learning from one example at a time (slow!), you show it32 examples together and update based on the average mistake. Way faster!

"A baby CALF learns to walk" → A model learns to predict

Connections

• 3.1.01-Gradient-Descent - TensorFlow computes gradients automatically
• 3.2.03-Backpropagation - What happens inside model.fit()
• 3.3.01-Neural-Network-Basics - The math behind Dense layers
• 3.07-Activation-Functions - Why we use ReLU, softmax
• 3.3.09-PyTorch-basics - Alternative framework (more Pythonic)
• 3.4.05-Overfitting-Regularization - Why we use Dropout, validation split
• 4.2.01-CNN-Architecture - Keras for image models (Conv2D layers)

#flashcards/ai-ml

What is a tensor in TensorFlow? :: A multi-dimensional array with uniform data type, immutable, that flows through computational graphs. Generalizes scalars (rank 0), vectors (rank 1), matrices (rank 2) to higher dimensions.

Concept Map

Hinglish (regional understanding)