Projects

Open-source projects to learn from

Free projects worth an evening each, from small classics that make one idea concrete to recent codebases that show the whole modern pipeline. For each: why it is worth reading, and the first file to open.

Classics

• BeginnerMIT★ 17.7K

  micrograd

  Python

  A complete automatic-differentiation engine in 94 lines, with a small neural-network library on top.

  Why: Backpropagation is usually taught as a formula and used as a black box. Here it is a Value class whose every operation records how to pass a gradient back, and a backward() that walks the graph in reverse topological order. Once you have read it, PyTorch's autograd is the same idea on tensors instead of scalars.

  Start with: micrograd/engine.py, then micrograd/nn.py (60 lines): look at how __mul__ stores the local derivative of each input, and why backward() must sort the graph before applying the chain rule.

  On this site: Backpropagation and Gradient Descent, What Is a Neural Network?

  GitHubOpen in Colab
  Run it here, in your browser

  micrograd: the autograd engine, checked against finite differences

  Run ▶

  Edit the codefrom math import isclose # micrograd's engine (MIT licence, (c) 2020 Andrej Karpathy, github.com/karpathy/micrograd), # with its comments trimmed and one-line helpers joined. Every gradient below comes from these lines. class Value: def __init__(self, data, _children=(), _op=''): self.data = data self.grad = 0 self._backward = lambda: None self._prev = set(_children) self._op = _op def __add__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data + other.data, (self, other), '+') def _backward(): self.grad += out.grad other.grad += out.grad out._backward = _backward return out def __mul__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data * other.data, (self, other), '*') def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad out._backward = _backward return out def __pow__(self, other): out = Value(self.data**other, (self,), f'**{other}') def _backward(): self.grad += (other * self.data**(other-1)) * out.grad out._backward = _backward return out def relu(self): out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU') def _backward(): self.grad += (out.data > 0) * out.grad out._backward = _backward return out def backward(self): topo, visited = [], set() def build_topo(v): if v not in visited: visited.add(v) for child in v._prev: build_topo(child) topo.append(v) build_topo(self) self.grad = 1 for v in reversed(topo): v._backward() def __neg__(self): return self * -1 def __radd__(self, other): return self + other def __sub__(self, other): return self + (-other) def __rsub__(self, other): return other + (-self) def __rmul__(self, other): return self * other def __truediv__(self, other): return self * other**-1 def __rtruediv__(self, other): return other * self**-1 # 1. The chain rule, checked. Change f and the numbers: the two columns must agree. def f(a, b): return (a * b + b**3).relu() / (a + 2) a, b = Value(1.5), Value(0.8) out = f(a, b) out.backward() h = 1e-6 num_a = (f(Value(1.5 + h), Value(0.8)).data - out.data) / h num_b = (f(Value(1.5), Value(0.8 + h)).data - out.data) / h print(f"f = {out.data:.6f}") print(f"df/da autograd {a.grad: .6f} finite difference {num_a: .6f}") print(f"df/db autograd {b.grad: .6f} finite difference {num_b: .6f}") print("agree:", isclose(a.grad, num_a, rel_tol=1e-4) and isclose(b.grad, num_b, rel_tol=1e-4)) # 2. Gradient descent on one neuron: learn y = 2x - 1 from four points. xs, ys = [-1.0, 0.0, 1.0, 2.0], [-3.0, -1.0, 1.0, 3.0] w, c = Value(0.0), Value(0.0) for step in range(60): loss = sum((w * x + c - y)**2 for x, y in zip(xs, ys)) / len(xs) w.grad, c.grad = 0, 0 loss.backward() w.data -= 0.1 * w.grad c.data -= 0.1 * c.grad if step % 20 == 0 or step == 59: print(f"step {step:2d} loss {loss.data:8.5f} w {w.data: .4f} c {c.data: .4f}")

  Runs in your browser. The first run downloads the Python runtime (~10 MB), then it is cached.

• BeginnerMIT★ 24.6K

  Neural Networks: Zero to Hero

  Python (PyTorch)

  The notebooks of a video course that builds language models from a scalar autograd engine upward, one lecture per step.

  Why: Each notebook builds the next model out of the previous one: micrograd, then a bigram character model, an MLP, batch normalisation, a backward pass written by hand, and a WaveNet-style network; the videos then continue to a GPT and its tokenizer. The progression is the point: nothing appears that the previous model did not need.

  Start with: lectures/micrograd, then lectures/makemore/makemore_part1_bigrams.ipynb: a language model that is only a table of counts, and the negative log-likelihood that tells you how good that table is.

  On this site: What Is a Neural Network?, Tokenization and Embeddings, The Transformer Architecture

  GitHubOpen in Colab
• BeginnerMIT★ 32.9K

  ML From Scratch

  Python (NumPy)

  Bare NumPy implementations of the classical algorithms: regressions, trees, boosting, SVMs, clustering, and a small deep-learning framework.

  Why: Libraries hide the loop that fits a model. Here every algorithm is one readable file, so you can see that ridge and lasso differ by one line of the regularisation term, or that gradient boosting is a loop that fits trees to residuals. It is no longer actively developed, which makes it stable to read, not something to deploy.

  Start with: mlfromscratch/supervised_learning/regression.py: one base class, and each regression (linear, polynomial, ridge, lasso, elastic net) differs only in its regularisation.

  On this site: Linear Regression from First Principles, Regularization: Ridge and Lasso, Decision Trees and Ensembles

  GitHub
• BeginnerApache-2.0★ 14.2K

  Hands-On Machine Learning (3rd edition) notebooks

  Python (scikit-learn, Keras)

  The notebooks behind Aurélien Géron's book, from a first end-to-end project to deep learning, each runnable in Colab.

  Why: Most tutorials start with a clean dataset. Chapter 2 starts with a messy housing table and walks through what a real project does before any model: a stratified test split, looking at the data, a preprocessing pipeline, then cross-validation and error analysis. That order is what beginners most often get wrong.

  Start with: 02_end_to_end_machine_learning_project.ipynb: watch where the test set is set aside, and that it is not touched again until the very end.

  On this site: What Is Statistical Learning?, Cross-Validation and Resampling

  GitHubOpen in Colab
• BeginnerMIT (code); text CC BY-NC-ND 3.0★ 50K

  Python Data Science Handbook

  Python (NumPy, pandas, Matplotlib, scikit-learn)

  Jake VanderPlas's book, full text as notebooks, free to read online: the working toolkit before the models.

  Why: Half of machine-learning work is NumPy broadcasting and pandas reshaping, and it is rarely taught on purpose. The book teaches it on purpose, then closes with a machine-learning part that explains the scikit-learn estimator interface and model validation clearly.

  Start with: notebooks/05.00-Machine-Learning.ipynb and the chapters after it: the estimator API, then hyperparameters and model validation.

  On this site: What Is Statistical Learning?, The Bias-Variance Tradeoff

  GitHubOpen in ColabGuide
• IntermediateMIT★ 28.2K

  Bayesian Methods for Hackers

  Python (PyMC)

  Bayesian inference taught through computation first and mathematics second, with PyMC.

  Why: Where a textbook derives a posterior, this book samples it and plots it, so the prior, the likelihood and the posterior are pictures before they are formulas. The first chapter infers when a person's texting behaviour changed, which is a change-point model most readers would not guess they could fit.

  Start with: Chapter1_Introduction/Ch1_Introduction_PyMC_current.ipynb: the texting example, and how the posterior of the switch day is read off the samples.

  On this site: Bayes' Theorem and Belief Updating, Learning the Numbers in a Probability Model

  GitHubOpen in Colab
• IntermediateBSD-3-Clause★ 67.4K

  scikit-learn

  Python (Cython)

  The reference library for classical machine learning, and one of the best-documented codebases to learn from.

  Why: Reading how a production library solves the problem you derived on paper shows what a derivation leaves out: which solver for which data shape, how the intercept is handled, what happens when the matrix is ill-conditioned. The docstrings cite the papers each choice comes from.

  Start with: sklearn/linear_model/_ridge.py (the solver chosen from the data shape), then _coordinate_descent.py for the lasso.

  On this site: Regularization: Ridge and Lasso, Linear Regression from First Principles

  GitHubGuide
• IntermediateMIT★ 12K

  Spinning Up in Deep RL

  Python (PyTorch, TensorFlow 1)

  OpenAI's educational resource for deep reinforcement learning: an introduction, a reading list, and short implementations of the key algorithms.

  Why: Its introduction to RL is one of the clearest short texts on policies, returns and value functions, and each algorithm (VPG, PPO, SAC and others) is a single file written to be read against its derivation. It is in maintenance mode, so treat the code as a textbook, not a library.

  Start with: The "Introduction to RL" pages on the site, then spinup/algos/pytorch/vpg/vpg.py: the policy-gradient estimator, its advantage estimate and its training loop in one file.

  On this site: Reinforcement Learning and Q-Learning, Markov Decision Processes

  GitHubGuide
• BeginnerMIT★ 12.6K

  Gymnasium

  Python

  The standard interface for reinforcement-learning environments, the maintained successor of OpenAI Gym.

  Why: Every RL algorithm talks to its world through the same two calls, reset and step. Learning that interface on a small environment, and reading how one is written, is the fastest way to run the Q-learning of the lessons on a real task.

  Start with: gymnasium/envs/toy_text/frozen_lake.py: a complete Markov decision process, with the transition probabilities written out as a table.

  On this site: Markov Decision Processes, Reinforcement Learning and Q-Learning

  GitHubGuide
• IntermediateMIT★ 67.5K

  Annotated deep-learning paper implementations

  Python (PyTorch)

  Short PyTorch implementations of 60+ papers, rendered on a website with the explanation beside each line.

  Why: Papers state an architecture in a paragraph and an equation; this shows the same thing as code with the paragraph next to it. It is the fastest way to check your reading of a paper against a working implementation.

  Start with: Multi-head attention (labml_nn/transformers/mha.py, annotated at nn.labml.ai/transformers/mha.html): the shapes of Q, K and V at every line.

  On this site: Attention and Self-Attention, The Transformer Architecture

  GitHubGuide
• BeginnerKaggle competition rules

  Titanic: Machine Learning from Disaster

  Python or R, in free Kaggle notebooks

  Kaggle's permanent beginner competition: predict who survived from a small passenger table, and get scored on held-out data.

  Why: It is small enough to understand every column and real enough to meet the classic traps: missing ages, categories to encode, and a validation score that flatters a model tuned against it. Kaggle's free Intro to Machine Learning course is the gentlest way in.

  Start with: Build a baseline that predicts "did not survive" for everyone (right for 549 of the 891 training passengers, 61.6%): every model after that has to beat it.

  On this site: Logistic Regression and Classification, Comparing Classifiers, and What Accuracy Hides, A Score That Loses to Doing Nothing

  KaggleGuide

Recent and worth it

• IntermediateApache-2.0★ 105.7K

  Build a Large Language Model (From Scratch)

  Python (PyTorch)

  The code of Sebastian Raschka's book: a GPT built, pretrained and fine-tuned step by step, chapter by chapter.

  Why: It covers the whole path in order (text to tokens, attention, the GPT block, pretraining, loading real GPT-2 weights, then fine-tuning for classification and for following instructions) with each chapter a notebook that runs on a laptop.

  Start with: ch02/01_main-chapter-code/ch02.ipynb (tokenisation and embeddings), then ch03 for attention built up from a single query.

  On this site: Tokenization and Embeddings, Attention and Self-Attention, The Transformer Architecture, Pretraining and Fine-Tuning

  GitHubOpen in Colab
• AdvancedMIT★ 58.3K

  nanochat

  Python (PyTorch)

  A minimal, complete pipeline for training a chat model: tokenizer, pretraining, supervised fine-tuning, reinforcement learning, evaluation and chat.

  Why: Its README reports a model at GPT-2 capability (about $43,000 to train in 2019) for about $48, some two hours on one 8xH100 node. You do not need the GPUs to learn from it: it is the whole modern pipeline in one small, readable codebase. It succeeds the same author's nanoGPT, whose own README now marks nanoGPT as deprecated.

  Start with: runs/speedrun.sh: the entire pipeline as one script, stage by stage, each line pointing at the file that implements it.

  On this site: Pretraining and Fine-Tuning, The Transformer Architecture

  GitHub
• AdvancedMIT★ 31.1K

  llm.c

  C / CUDA

  GPT-2 training in plain C and CUDA, with no PyTorch and no Python.

  Why: Without a framework, every forward and backward pass is written out by hand: layer norm, attention, the softmax gradient, AdamW. It is the most concrete answer to "what does the framework actually compute?".

  Start with: train_gpt2.c, the CPU reference (about 1,200 lines): find layernorm_forward and layernorm_backward and check them against the derivation.

  On this site: Backpropagation and Gradient Descent, The Transformer Architecture

  GitHub
• IntermediateMIT★ 129.8K

  llama.cpp

  C / C++

  Runs open-weight language models locally, on a laptop CPU or GPU, with quantised weights.

  Why: It is how most people run an open model on their own machine, and it makes quantisation tangible: the same model at 16, 8 and 4 bits per weight, and what that does to memory, speed and answers.

  Start with: docs/build.md to build it, then run a small quantised model and compare two quantisation levels on the same prompt.

  On this site: Pretraining and Fine-Tuning

  GitHub
• IntermediateApache-2.0★ 76.9K

  Unsloth

  Python (PyTorch)

  Fine-tuning of open models with far less GPU memory, with ready notebooks that run on a free Colab GPU.

  Why: Fine-tuning is usually out of reach without a large GPU. Its notebooks fine-tune a small open model with LoRA on a free Colab T4, which turns the fine-tuning lesson into something you can run in an afternoon.

  Start with: The notebook list in unslothai/notebooks: pick a small model, and read the LoRA settings (rank, which layers) before you train.

  On this site: Pretraining and Fine-Tuning

  GitHubGuide
• AdvancedApache-2.0★ 27.8K

  LeRobot

  Python (PyTorch)

  Hugging Face's library for learning robot policies from demonstrations, with datasets, pretrained policies and simulation.

  Why: It is where the learning ideas of the site meet a physical system: imitation learning from recorded demonstrations and reinforcement learning, on low-cost arms and in simulation, with shared datasets on the Hub.

  Start with: The documentation's getting-started pages, then load a dataset from the Hub and look at what one demonstration contains.

  On this site: Reinforcement Learning and Q-Learning

  GitHubGuide