Foundations of Neural Networks and Transformers · Lesson 2
Other Mathematical Topics: Learning with Derivatives
Learning means minimising a loss function that measures error. Mean squared error suits regression; cross-entropy suits classification.
Probability matters too: softmax turns raw scores into probabilities that sum to 1, and models are trained to give high probability to correct answers.
Derivatives of the loss with respect to each weight tell us how to nudge that weight to reduce error — this is the core idea behind all training.
Key points
- Loss functions measure error
- MSE for regression, cross-entropy for classification
- Softmax converts scores to probabilities
- Derivatives guide weight updates