3.3.8 · D5Deep Learning Frameworks
Question bank — TensorFlow - Keras basics
This is a rapid-fire misconception check for the parent note. Each line hides its answer — think first, then reveal. Every answer gives you the reasoning, not just a verdict. If a term feels unfamiliar, the parent note and its prerequisites (3.3.01-Neural-Network-Basics, 3.07-Activation-Functions, 3.1.01-Gradient-Descent) build them from zero.
True or false — justify
A tensor and a NumPy array are the same thing.
False in spirit: a
tf.constant tensor is immutable and can live on a GPU and be tracked by the autodiff graph, whereas a NumPy array is mutable CPU memory. They interconvert, but the immutability is what lets TensorFlow compile and parallelize safely.Stacking two Dense layers with no activation gives a more powerful model than one Dense layer.
False. Composing two linear maps then equals one linear map , so without a nonlinearity like ReLU the depth collapses to a single layer — see 3.07-Activation-Functions.
Dropout(0.2) slows the model down at prediction time because it keeps dropping neurons.
False. Dropout is active only during training; at inference Keras passes everything through untouched (the scaling was already applied during training via inverted dropout).
A higher learning rate always makes training faster.
False. Too large an overshoots the minimum and the loss diverges; the Taylor argument only guarantees a decrease for small . See 3.1.01-Gradient-Descent.
softmax and sigmoid are interchangeable for a 10-class classifier.
False. Softmax couples all 10 logits so they sum to 1 (one distribution over classes); 10 independent sigmoids allow each class to be "on" independently, which is multi-label, not single-label classification.
The Functional API can build networks the Sequential API cannot.
True. Sequential is a single straight chain (one input, one output per layer); the Functional API allows branches, merges, multiple inputs/outputs and skip connections like those in 4.2.01-CNN-Architecture.
SparseCategoricalCrossentropy and CategoricalCrossentropy compute different mathematical losses.
False on the loss value: they compute the same cross-entropy. They differ only in label format — sparse takes integer labels like
7, categorical takes one-hot vectors like [0,...,1,...,0].Normalizing pixels by dividing by 255 changes what the network can learn.
False about capacity, true about training. It does not change the function class, but rescaling inputs to keeps gradients well-scaled so gradient descent converges faster and more stably.
One epoch and one batch are the same unit of training.
False. A batch is one weight update on
batch_size examples; an epoch is a full pass over the whole training set, i.e. many batches.Adding more epochs can only help — never hurts.
False. Past a point the model keeps fitting training noise and validation loss rises; that is overfitting, which
validation_split exists to reveal.Spot the error
keras.layers.Dense(10, activation='softmax', input_shape=(784,)) placed as the first and only layer for MNIST — what's wrong conceptually?
Nothing crashes, but a single softmax-over-linear layer is just logistic regression: no hidden ReLU layer means no nonlinear feature learning, so accuracy is capped far below a real network.
model.fit(x_train, y_train) is called before model.compile(...).
Error:
fit needs an optimizer, loss and metrics to know how to update weights and what to minimize — compile must run first to attach them.Using loss='categorical_crossentropy' with integer labels y_train = [3, 7, 1, ...].
Shape/format mismatch: categorical crossentropy expects one-hot targets. Either one-hot the labels or switch to
sparse_categorical_crossentropy.Feeding raw (28, 28) images straight into Dense(128).
A
Dense layer expects a flat vector; the images must be reshaped to (784,) (or a Flatten layer inserted) or the shapes won't align.In a subclassed model, calling self.dropout(x) without passing training=training.
Dropout can't tell train from inference, so it may drop neurons during prediction (or never during training), corrupting results. Always thread the
training flag through call.Dense(128) followed immediately by another Dense(10) with both using activation='softmax'.
Softmax on a hidden layer forces its 128 outputs to sum to 1, throttling the signal and information flow — softmax belongs only on the final classification layer; hidden layers should use ReLU.
Setting input_shape=(784,) on every layer of a Sequential model.
Only the first layer needs
input_shape; Keras infers all later shapes from the graph. Specifying it repeatedly is redundant and can conflict.Why questions
Why must the first layer know input_shape but later layers do not?
Keras infers each layer's input size from the previous layer's output, but the very first layer has no predecessor, so you must tell it the raw feature dimension to build its weight matrix.
Why does inverted dropout divide the kept activations by ?
To keep the expected value unchanged: . This way the layer's average signal magnitude is the same in training and inference, so no rescaling is needed at test time.
Why is Adam often preferred over plain gradient descent as a default?
Adam adapts a per-parameter learning rate by combining momentum (smoothed gradient) and RMSprop (gradient magnitude scaling), so it handles poorly-scaled or sparse gradients with little tuning.
Why does softmax use the exponential rather than, say, just the raw logits?
The exponential guarantees all outputs are positive and preserves ordering, while the normalizing sum makes them add to 1 — turning arbitrary real logits into a valid probability distribution.
Why does a large batch_size give smoother but sometimes worse-generalizing gradients?
Averaging over many examples reduces the noise in each gradient estimate (smoother path), but that same noise in small batches acts as mild regularization that can help escape sharp minima.
Why does TensorFlow make tensors immutable?
Immutability means operations have no side effects, so the computation graph can be safely reordered, cached, compiled, and parallelized across devices for automatic differentiation.
Why does model subclassing give more control than the Sequential or Functional API?
You write the forward pass yourself in
call, allowing data-dependent (dynamic) control flow — if/loops, conditional dropout, custom training logic — that a static layer list cannot express.Edge cases
What happens if validation_split=0.0?
No data is held out, so you get no validation loss curve and no early warning of overfitting — training metrics alone can look great while the model memorizes.
What if the final Dense layer has more units than the number of classes?
Softmax still produces a valid distribution, but over the wrong number of categories; predictions won't align with your label indices and the loss will be ill-defined against integer targets.
What does Dropout(0.0) do?
It drops nothing — an identity layer. Useful as a placeholder or to disable regularization without changing the architecture.
What does Dropout(1.0) do, and why is it a trap?
It zeros every activation, so the layer outputs all zeros and the network learns nothing downstream — the scaling even divides by zero. Never use .
A tensor of shape () versus (1,) — are they the same?
No.
() is a rank-0 scalar (a lone number); (1,) is a rank-1 vector holding one element. Many broadcasting bugs come from confusing them.If every input feature is already identical across examples, what can the network learn?
Nothing discriminative — with no variation to key off, the best it can do is predict the class prior; this is a degenerate data problem no architecture fixes.
Recall Quick self-test
Give one reason softmax belongs only on the output layer. ::: Forcing hidden activations to sum to 1 destroys the free signal magnitude that later layers need.
State the one thing the first Sequential layer needs that others don't. ::: input_shape — the raw feature dimension.
Next stop: contrast this API with 3.3.09-PyTorch-basics to see the same ideas under a dynamic-graph philosophy.