Foundations of Neural Networks and Transformers · Lesson 7
Neural Networks for Text
Text is tokenised and each token mapped to an embedding vector. Recurrent networks (RNN, LSTM, GRU) read tokens in order with a hidden state, but struggle with long-range dependencies and cannot parallelise well.
Attention lets every token look at every other token and weigh their relevance. The Transformer uses self-attention with queries, keys and values, multi-head attention, positional encodings and feed-forward layers.
Encoder models (BERT) suit understanding tasks; decoder-only models (GPT, Claude, Llama) generate text; encoder-decoder models (T5) map input to output.
Key points
- Tokens → embeddings
- RNNs/LSTMs process sequentially
- Self-attention: queries, keys, values
- Encoder, decoder and encoder-decoder families