Foundations of Neural Networks and Transformers · Lesson 1
Linear Algebra; Calculus
Neural networks are built from linear algebra. Data is stored as vectors and matrices; a layer multiplies an input vector by a weight matrix and adds a bias. The dot product measures how aligned two vectors are.
Calculus tells us how outputs change when inputs change. A derivative is the rate of change; a partial derivative changes one variable at a time; the gradient collects all partial derivatives into a vector pointing in the direction of steepest increase.
The chain rule lets us differentiate functions inside functions — exactly the structure of stacked network layers.
Key points
- Vectors, matrices and matrix multiplication
- Dot product = alignment
- Gradient = vector of partial derivatives
- Chain rule handles nested functions