Foundations of Neural Networks and Transformers · Lesson 3
Foundations of NN: Gradient Descent
Gradient descent repeatedly updates each weight by subtracting the learning rate times the gradient: w = w − η·∂L/∂w.
The learning rate controls step size. Too large and training diverges; too small and it crawls. Stochastic and mini-batch gradient descent use small batches of data per step, which is faster and adds helpful noise.
Optimisers such as Momentum, RMSProp and Adam adapt the steps for faster, more stable training.
Key points
- Update rule: w ← w − η·gradient
- Learning rate sets step size
- Mini-batches balance speed and stability
- Adam is the common default optimiser
w, lr = 0.0, 0.1
for step in range(100):
grad = 2 * (w - 3) # derivative of (w-3)^2
w = w - lr * grad
print(w) # approaches 3