Foundations of Neural Networks and Transformers · Lesson 4
Building a Neural Network
A neural network stacks layers of neurons. Each neuron computes a weighted sum of inputs plus a bias, then applies an activation function such as ReLU, sigmoid or tanh.
Without non-linear activations, stacked layers collapse into one linear function, so activations are what let networks learn complex patterns.
In PyTorch you define layers, a loss and an optimiser, then loop: forward pass, compute loss, backward pass, optimiser step.
Key points
- Neuron = weighted sum + bias + activation
- Non-linearity enables complex functions
- Input, hidden and output layers
- Forward → loss → backward → step
import torch.nn as nn, torch
model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 3))
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for x, y in loader:
loss = loss_fn(model(x), y)
opt.zero_grad(); loss.backward(); opt.step()