Training Deep Networks

The training loop, again

You have a model, a loss and gradients from backprop. Training is the loop around them:

1. Take a small chunk of the data, called a mini-batch.
2. Forward pass: compute predictions and the loss on that batch.
3. Backward pass: compute gradients.
4. Optimizer step: update the weights.
5. Repeat. One full pass over the data is an epoch.

Why mini-batches instead of the whole dataset or one example at a time?

• The whole dataset per step is slow and often does not fit in memory.
• One example per step gives a noisy gradient and wastes hardware that is built for parallel work.
• A batch of 32 to 512 gives a decent gradient estimate at a fair cost. The noise even helps the model escape poor spots.

This is SGD (stochastic gradient descent): gradient descent where each gradient comes from a random mini-batch.