Natural Language Processing
Making language computable: representation, sequence models, attention, and the language models that followed from them.
Learning paths (1)
Encyclopedia (3)
Tokenization
The process of splitting text into the discrete units a language model actually operates on, typically subword fragments rather than whole words.
Self-Attention
A mechanism that lets every position in a sequence attend to every other, computing each output as a weighted sum of values whose weights come from query-key similarity.
Transformer
A neural architecture built on stacked self-attention and feed-forward layers, which replaced recurrence as the standard for sequence modelling.
Articles (3)
Tokenization and Embeddings
How text becomes numbers a model can train on: building a vocabulary, why byte pair encoding never needs an unknown token, the embedding layer as a lookup that is provably one-hot times a matrix, and why position has to be added back in by hand.
The Transformer Architecture
Assembling a GPT from attention: multi-head projections, layer normalization worked by hand, why shortcut connections rescue the gradient, the 4x feed-forward expansion, and a parameter count that reproduces GPT-2 small at 124 million exactly.
Attention and Self-Attention
Queries, keys, and values built from the ground up: why attention exists, how scaled dot-product attention is computed, why it is divided by the square root of the dimension, and how causal masking works, with every matrix computed and checked.
Tools (1)
Datasets (1)
Research (4)
Neural Machine Translation of Rare Words with Subword Units
Adapts byte pair encoding to text segmentation, building a subword vocabulary by repeatedly merging the most frequent adjacent symbol pair, so that rare words decompose into known fragments.
Attention Is All You Need
Introduces the transformer, an architecture built entirely from attention and feed-forward layers with no recurrence, originally developed for machine translation.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Pretrains a transformer encoder to predict masked tokens using context from both directions, then fine-tunes the same model on downstream tasks with a small task-specific head.
Language Models are Few-Shot Learners
Describes GPT-3 and shows that a sufficiently large decoder-only language model can perform new tasks from a handful of examples supplied in its prompt, with no gradient updates.