Machine Learning Curriculum

Curriculum pages: Curated Bookmarks · Personal

The rule for prerequisites

Use mathematics just in time. When a concept blocks the current module, revise only that concept, apply it, and continue. Repeated contact is part of learning—not evidence that you are behind.

How to use this curriculum

Each phase has three layers:

1. Learn — understand the central ideas and vocabulary.
2. Implement — reproduce a small method without hiding everything behind a library.
3. Explain — write a short note about what worked, what failed, and why.

Work through Phases 0–2 in order. After that, choose one research branch and one engineering branch. Treat the remaining subjects as a reference map rather than a single enormous checklist.

30-Day Book and Paper Sprint

Goal: build a coherent first picture of learning theory, linear algebra, and the architectural lineage of modern AI in one intensive month.

This is a selected-reading plan, not a promise to finish three textbooks and thirty papers cover to cover. Budget roughly 12–15 focused hours per week. If time is limited, complete the items marked Core and move the rest to the long-term curriculum.

The book stack

Weekly rhythm

• Monday–Thursday: 60 minutes on the primary book, 45 minutes on the companion chapter, and 30 minutes on exercises or derivations
• Friday: read one paper and produce a one-page paper card
• Saturday: implement or derive the week’s central idea; then review weak points
• Sunday: catch up or rest—do not begin new material

Week 1 — What is learning?

Core reading

• Learning From Data, Chapter 1 — the learning problem, feasibility, error, and noise
• Understanding Machine Learning, Chapters 2–3 — the statistical learning framework and PAC model
• Linear Algebra Done Right, Sections 1A–1C and 2A–2C — fields, vector spaces, span, independence, bases, and dimension
• Paper: ImageNet Classification with Deep Convolutional Neural Networks

Output

• Write one learning problem as domain, data distribution, hypothesis class, loss, and success criterion
• Solve at least six Axler exercises and explain span, independence, basis, and dimension without notes
• Create the first paper card using the template below

Week 2 — Why does learning generalize?

Core reading

• Learning From Data, Chapter 2 — VC dimension, generalization bounds, bias, and variance
• Understanding Machine Learning, Chapters 4–6 — uniform convergence, the bias–complexity tradeoff, and VC dimension
• Linear Algebra Done Right, Sections 3A–3D — linear maps, null spaces, ranges, matrices, and invertibility
• Paper: Deep Residual Learning for Image Recognition

Stretch

• Understanding Machine Learning, Chapter 7 — structural risk minimization and minimum description length
• Paper: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights from the Top 30 reading list

Output

• Derive one finite-class generalization bound and explain what changes as the hypothesis class grows
• Draw the relationship between injectivity, null space, surjectivity, and range
• Compare plain depth with residual connections in a short architecture note

Week 3 — Models and optimization

Core reading

• Learning From Data, Chapter 3 — classification, linear and logistic regression, gradient descent, and nonlinear transforms
• Understanding Machine Learning, Chapters 9, 12, and 14 — linear predictors, convex learning, and stochastic gradient descent
• Linear Algebra Done Right, Sections 5A and 6A–6C — eigenvalues, inner products, orthogonality, minimization, and pseudoinverses
• Paper: Neural Machine Translation by Jointly Learning to Align and Translate
• Paper: Attention Is All You Need

Stretch

• Understanding Machine Learning, Chapter 15 — support vector machines

Output

• Implement linear and logistic regression without a model-fitting library
• Derive the gradient of logistic loss and connect least squares to orthogonal projection
• Explain, with matrix shapes, how Bahdanau attention leads toward scaled dot-product attention

Week 4 — Overfitting, selection, and scale

Core reading

• Learning From Data, Chapters 4–5 — overfitting, regularization, validation, Occam’s razor, sampling bias, and data snooping
• Understanding Machine Learning, Chapters 11 and 13 — model selection, validation, regularization, and stability
• Linear Algebra Done Right, Sections 7B and 7E — the spectral theorem and singular value decomposition
• Paper: Scaling Laws for Neural Language Models

Stretch

• Paper: GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
• Understanding Machine Learning, Chapter 20 — scan the neural-networks chapter and list which assumptions no longer match modern deep learning practice

Output

• Run an overfitting experiment that varies data size, model capacity, and regularization strength
• Use SVD for a low-rank approximation and plot reconstruction error against retained rank
• Write a two-page synthesis: generalization, representation, optimization, and scale

Paper-reading method

Use Ilya Sutskever’s Top 30 reading list as the source queue. Do not treat the summaries as substitutes for the original papers.

For the first pass, read in this order:

1. Abstract and conclusion
2. Introduction and figures
3. Method and central equations
4. Experiments and ablations
5. Related work only after you understand the paper’s claim

Create a paper card with:

• Problem: What limitation existed before this work?
• Claim: What is the paper’s central claim in one sentence?
• Mechanism: What changed mathematically or architecturally?
• Evidence: Which experiment most strongly supports the claim?
• Assumption: What must be true for the result to matter?
• Weakness: What is untested, expensive, or no longer convincing?
• Connection: Which book concept makes the paper easier to understand?
• Reproduction: What is the smallest result you could reproduce?

End-of-month completion standard

• Finish all five chapters of Learning From Data
• Complete the selected theory and linear-algebra sections marked Core
• Solve at least 25 book exercises across the month
• Produce at least five paper cards
• Implement linear regression, logistic regression, and one SVD experiment
• Explain PAC learning, VC dimension, regularization, SGD, attention, residual learning, and scaling without relying on memorized definitions
• Choose the next Phase 1 topic from the gaps revealed by the final synthesis

Phase 0 — Working Foundations

Purpose: become comfortable enough with code, data, and mathematical notation to learn by building.

Programming and algorithms

• Write clear Python and use NumPy, pandas, Matplotlib, and notebooks
• Understand arrays, hash tables, trees, graphs, sorting, and searching
• Analyze runtime and memory with Big-O notation
• Use Git, environments, tests, and reproducible project structure
• Implement one data-processing pipeline from raw input to a checked dataset

Linear algebra and calculus

• Vectors, matrices, matrix multiplication, norms, and projections
• Eigenvalues, eigendecomposition, and singular value decomposition
• Partial derivatives, gradients, Jacobians, and Hessians
• Multivariable chain rule and Taylor approximation
• Implement linear regression and gradient descent with NumPy

Probability and statistics

• Random variables, common distributions, expectation, and variance
• Conditional probability, Bayes’ rule, independence, and correlation
• Sampling, estimation, confidence intervals, and hypothesis tests
• Law of large numbers and central limit theorem
• Markov, Chebyshev, Hoeffding, and Chernoff bounds at intuition level

Resources already selected

• Introduction to Statistics and Data Analysis — Christian Heumann, Michael Schomaker, Shalabh
• All of Statistics: A Concise Course in Statistical Inference — Springer Texts in Statistics
• Intro to Inferential Statistics
• Descriptive Statistics
• Foundations of Data Science — use the probability and concentration material as needed
• https://seeing-theory.brown.edu/

Completion evidence

• A NumPy notebook implementing linear regression, gradient descent, and uncertainty estimates
• A short note explaining how conditioning, learning rate, and sample size affected the result

Phase 1 — Core Machine Learning

Purpose: learn the statistical and computational ideas shared by most ML systems.

Supervised and unsupervised learning

• Linear and logistic regression
• Decision trees, ensembles, nearest neighbors, and support vector machines
• Clustering, principal component analysis, and matrix factorization
• Loss functions, regularization, bias–variance tradeoff, and data leakage
• Train/validation/test splits, cross-validation, calibration, and error analysis

Optimization

• Convex sets and functions; why a local optimum can be global
• Gradient descent, stochastic gradient descent, momentum, and Adam
• Linear programs, duality, and the meaning of a dual certificate
• Constraints, Lagrangians, and KKT conditions
• Numerical stability, conditioning, and floating-point limitations

Resources

• Convex Optimization — Boyd and Vandenberghe — begin with Chapters 1–3; return to duality when needed
• Algorithms — Jeff Erickson — use the complexity, NP-hardness, and approximation chapters when they become relevant

Learning theory

• Empirical risk minimization and Bayes-optimal prediction
• PAC learning, VC dimension, and sample complexity
• Uniform convergence and algorithmic stability
• Rademacher complexity and generalization bounds
• Statistical decision theory and minimax estimation

Completion evidence

• Compare at least three model families on one real dataset
• Include a baseline, ablation, error analysis, and reproducible evaluation
• Explain whether the main limitation came from data, optimization, capacity, or evaluation

Phase 2 — Deep Learning Fundamentals

Purpose: understand the components from which modern foundation models are built.

Neural-network mechanics

• Perceptrons, multilayer networks, activations, and loss functions
• Backpropagation and reverse-mode automatic differentiation
• Initialization, normalization, regularization, and optimization
• Convolutional networks and residual connections
• Recurrent networks, LSTMs, GRUs, and vanishing gradients
• Attention, multi-head attention, positional encoding, and transformers

Representation and generative learning

• Transfer learning and the pretrain–fine-tune paradigm
• Self-supervised, contrastive, and representation learning
• Autoencoders and variational autoencoders
• Generative adversarial networks
• Normalizing flows
• Score matching and diffusion models

Completion evidence

• Implement backpropagation for a small network without autograd
• Train one vision or language model using a modern framework
• Reproduce a small result from a paper and document deviations from the original

Phase 3 — Choose a Research Branch

Choose one primary branch. Add a second only after completing a substantial project in the first.

A. Language, foundation models, and agents

• Tokenization and autoregressive language modeling
• BERT/GPT objectives, instruction tuning, and preference optimization
• Scaling laws, emergent behavior, and model failure modes
• Retrieval, tool use, memory, and agentic workflows
• Multimodal and vision-language models
• Mechanistic interpretability, sparse autoencoders, and evaluation
• Alignment: reward misspecification, scalable oversight, and Goodhart’s law

Build: an evaluated tool-using or retrieval-augmented system with traces, failure categories, and an ablation.

B. Computer vision and geometric learning

• Convolutional architectures and vision transformers
• Detection, segmentation, and representation learning
• Contrastive image–text learning and multimodal grounding
• Graph neural networks and message passing
• Equivariance, invariance, and geometric deep learning
• Manifold learning and differential geometry
• Neural ODEs, physics-informed models, and scientific ML

Build: a vision or geometric-learning system tested against distribution shift, not only a random validation split.

C. Reinforcement learning and decision making

• Markov decision processes and Bellman equations
• Dynamic programming, Q-learning, and SARSA
• Policy gradients and variance reduction
• Actor–critic methods, PPO, and trust regions
• Model-based RL, planning, world models, and MCTS
• Multi-agent RL and emergent communication
• Continual learning, meta-learning, and few-shot adaptation

Build: an RL agent with learning curves across multiple seeds and a discussion of reward design and sample efficiency.

D. Probabilistic modeling and causality

• Graphical models, d-separation, and factor graphs
• Bayesian inference and hierarchical models
• Variational inference and the ELBO
• Importance sampling, MCMC, and convergence diagnostics
• Markov chains, mixing, and message passing
• Potential outcomes and observational study design
• Structural causal models, interventions, and do-calculus
• Causal representation learning

Build: a probabilistic or causal analysis that states its assumptions, checks diagnostics, and distinguishes prediction from intervention.

E. Theory, optimization, and privacy

• Online learning, regret, bandits, UCB, Thompson sampling, and EXP3
• Kernel methods and reproducing-kernel Hilbert spaces
• Sparse methods and compressed sensing
• Spectral methods and random matrix theory
• Approximation theory and neural-network expressivity
• Differential privacy and composition
• Dynamical systems, stability, and optimal control
• Computational hardness of learning and inference

Build: a theorem-led study that pairs a derivation or proof with an experiment illustrating where its assumptions matter.

F. Algorithmic game theory and mechanism design

• Strategic-form, zero-sum, Bayesian, and extensive-form games
• Mixed strategies, best responses, Nash and correlated equilibria
• Minimax, linear-program duality, and equilibrium existence
• No-regret learning and Multiplicative Weights
• Complexity of equilibrium computation, total search, and PPAD
• Congestion and potential games; price of anarchy
• Counterfactual regret minimization and Monte-Carlo CFR
• Auctions, truthful mechanisms, VCG, and computational constraints
• Multi-agent learning and exposure-game equilibria

Build: implement one equilibrium or regret-minimization algorithm, test it on several games, and explain both its guarantee and its failure modes.

Algorithmic Game Theory — Just-in-Time Math Toolkit

Do not study this entire table in advance. Use it as a lookup when a game-theory topic becomes blocked.

After each revision, record: what blocked you, what you revised, and where you used it immediately.

Phase 4 — Computing and Engineering for ML

Study this alongside a research branch. Start with the common production path, then go lower-level only when scale or performance creates a real need.

Common engineering path

• Data formats, loaders, preprocessing, validation, and feature pipelines
• Experiment tracking, configuration, reproducibility, and hyperparameter search
• Model versioning, registries, deployment, and rollback
• Inference batching, caching, observability, and load testing
• A/B tests, canary deployments, drift monitoring, and incident response
• Edge deployment, distillation, pruning, and quantization
• Compute efficiency, cost measurement, and Green AI

Scaling and systems path

• GPU execution, memory hierarchy, and CUDA kernels
• Mixed precision, activation checkpointing, and memory-efficient attention
• Data, tensor, pipeline, and optimizer-state parallelism
• Collective communication, NCCL, RDMA, and network topology
• Automatic differentiation systems and ML compilers
• Profiling with a roofline-style performance model
• Fault tolerance and distributed checkpointing
• TPU and other accelerator architectures

Completion evidence

• Deploy a model behind an API with tests and monitoring
• Measure latency, throughput, memory, cost, and model quality
• Identify one bottleneck with a profiler and verify the improvement after changing it

Phase 5 — Advanced Mathematical Reference

These subjects deepen specific research directions. They are not universal prerequisites.

• Measure theory and Lebesgue integration — rigorous probability and theoretical ML
• Information theory — representation, compression, generalization, and communication
• Numerical linear algebra — large models, scientific ML, and stable solvers
• Functional analysis and RKHS theory — kernels and nonparametric learning
• Random matrix theory — high-dimensional statistics and spectral methods
• Differential geometry — manifolds, information geometry, and equivariant learning
• Tensor decomposition — latent-factor, multimodal, and scientific models
• Dynamical systems and control — optimization dynamics, robotics, and continuous-time models

Progress checkpoints

Use these checkpoints to decide whether to advance. Time spent is not the test; retained capability is.

Learning log template

For every substantial module or project, capture:

• Question: What am I trying to understand or build?
• Prediction: What do I expect, and why?
• Evidence: What did the experiment, derivation, or source show?
• Failure: What did not work?
• Just-in-time revision: Which prerequisite did I revisit?
• Next step: What is the smallest useful follow-up?