Foundations of Neural Networks and Transformers · Lesson 6
Neural Networks Advanced
Overfitting happens when a model memorises training data and fails on new data. Fight it with more data, dropout, weight decay, data augmentation and early stopping.
Batch normalisation and layer normalisation stabilise training. Good weight initialisation and residual (skip) connections help very deep networks train.
Split data into training, validation and test sets; tune on validation and report on test only once.
Key points
- Overfitting: great on train, poor on new data
- Dropout, weight decay, augmentation, early stopping
- Normalisation and residuals enable depth
- Train / validation / test splits