Sutskever 30 - Complete Implementation Suite

Comprehensive toy implementations of the 30 foundational papers recommended by Ilya Sutskever

Buy the Google Colab Code for this Repo at Gumroad

Overview

This repository contains detailed, educational implementations of the papers from Ilya Sutskever's famous reading list - the collection he told John Carmack would teach you "90% of what matters" in deep learning.

Progress: 30/30 papers (100%) - COMPLETE! 🎉

Each implementation:

✅ Uses only NumPy (no deep learning frameworks) for educational clarity
✅ Includes synthetic/bootstrapped data for immediate execution
✅ Provides extensive visualizations and explanations
✅ Demonstrates core concepts from each paper
✅ Runs in Jupyter notebooks for interactive learning

Quick Start

# Navigate to the directory
cd sutskever-30-implementations

# Install dependencies
pip install numpy matplotlib scipy

# Run any notebook
jupyter notebook 02_char_rnn_karpathy.ipynb

The Sutskever 30 Papers

Foundational Concepts (Papers 1-5)

#	Paper	Notebook	Key Concepts
1	The First Law of Complexodynamics	✅ `01_complexity_dynamics.ipynb`	Entropy, Complexity Growth, Cellular Automata
2	The Unreasonable Effectiveness of RNNs	✅ `02_char_rnn_karpathy.ipynb`	Character-level models, RNN basics, Text generation
3	Understanding LSTM Networks	✅ `03_lstm_understanding.ipynb`	Gates, Long-term memory, Gradient flow
4	RNN Regularization	✅ `04_rnn_regularization.ipynb`	Dropout for sequences, Variational dropout
5	Keeping Neural Networks Simple	✅ `05_neural_network_pruning.ipynb`	MDL principle, Weight pruning, 90%+ sparsity

Architectures & Mechanisms (Papers 6-15)

#	Paper	Notebook	Key Concepts
6	Pointer Networks	✅ `06_pointer_networks.ipynb`	Attention as pointer, Combinatorial problems
7	ImageNet/AlexNet	✅ `07_alexnet_cnn.ipynb`	CNNs, Convolution, Data augmentation
8	Order Matters: Seq2Seq for Sets	✅ `08_seq2seq_for_sets.ipynb`	Set encoding, Permutation invariance, Attention pooling
9	GPipe	✅ `09_gpipe.ipynb`