Overfitting and Regularization
Why a model that scores perfectly on its training data can fail on new data, and the standard ways to prevent it.
Intuition: memorising versus learning
A model is trained on examples but used on examples it has never seen. Generalisation is how well it does on those new examples.
- Overfitting: the model memorises quirks and noise in the training set. Training error is tiny, new-data error is large.
- Underfitting: the model is too simple to capture the real pattern. Both errors are large.
- Noise is the random part of the data that no model can predict. A model that fits the noise is learning something that will not repeat.
| Situation | Train error | Validation error | Usual cause |
|---|---|---|---|
| Underfit | High | High (similar) | Model too simple, too few features |
| Good fit | Low | Slightly higher | - |
| Overfit | Very low | Much higher | Model too flexible for the amount of data |
Quick check
A model has 2% error on training data and 30% error on a held-out validation set. What is the most likely problem?
1 of 5