Foundational AI/ML Papers

Papers that defined the field. Ordered chronologically by era.

Era 1 — Origins (1943–1970)

Year Title Authors Venue Contribution
1943 A Logical Calculus of Ideas Immanent in Nervous Activity McCulloch & Pitts Bull. Math. Biophysics First mathematical model of an artificial neuron
1950 Computing Machinery and Intelligence Alan Turing Mind Turing test; "Can machines think?"
1958 The Perceptron Frank Rosenblatt Psych. Review First neural learning algorithm

Era 2 — Backpropagation and Foundations (1986–1998)

Year Title Authors Venue Contribution
1986 Learning Representations by Back-Propagating Errors Rumelhart, Hinton, Williams Nature Backpropagation — trains multilayer networks
1997 Long Short-Term Memory Hochreiter & Schmidhuber Neural Computation LSTM — solves vanishing gradient in sequences
1998 Gradient-Based Learning Applied to Document Recognition LeCun et al. Proc. IEEE LeNet-5 — CNN for digit recognition

Era 3 — Deep Learning Renaissance (2006–2012)

Year Title Authors Venue arXiv Contribution
2006 A Fast Learning Algorithm for Deep Belief Nets Hinton, Osindski, Teh Neural Computation — Deep Belief Networks; catalyzed the revolution
2010 Rectified Linear Units Improve RBMs Nair & Hinton ICML — ReLU — modern activation standard
2012 ImageNet Classification with Deep CNNs (AlexNet) Krizhevsky, Sutskever, Hinton NIPS — Won ImageNet by a huge margin; started the DL era
2014 Dropout Srivastava et al. JMLR — Regularization via random deactivation
2014 Batch Normalization Ioffe & Szegedy ICML 1502.03167 Normalization between layers; speeds up training

Era 4 — Deep CNNs and Detection (2014–2016)

Year Title Authors arXiv Contribution
2014 VGGNet Simonyan & Zisserman ICLR — Depth with 3×3 convolutions
2015 ResNet He et al. CVPR 1512.03385 Skip connections — networks with 152+ layers
2014 R-CNN Girshick et al. CVPR 1311.2524 Region-based detection
2015 Faster R-CNN Ren et al. NIPS 1506.01497 Region Proposal Network; real-time detection
2016 YOLO Redmon et al. CVPR 1506.02640 Single-pass detection; 45 FPS

Era 5 — Sequences, Attention and Embeddings (2013–2017)

Year Title Authors arXiv Contribution
2013 Word2Vec Mikolov et al. ICLR — Efficient word embeddings
2014 GloVe Pennington, Socher, Manning EMNLP — Global matrix factorization + local context
2014 Seq2Seq Sutskever, Vinyals, Le NIPS 1409.3215 Encoder-decoder with LSTMs; basis of NMT
2014 Neural MT with Attention Bahdanau, Cho, Bengio ICLR 1409.0473 Attention mechanism — the model focuses on what is relevant
2018 ELMo Peters et al. NAACL 1802.05365 Bidirectional contextualized embeddings

Era 6 — Generative Models (2013–2020)

Year Title Authors arXiv Contribution
2013 VAE Kingma & Welling ICLR 1312.6114 Probabilistic variational autoencoders
2014 GAN Goodfellow et al. NIPS 1406.2661 Adversarial networks — realistic generation
2015 DCGAN Radford, Metz, Chintala ICLR 1511.06434 Practical convolutional GANs for images
2020 DDPM Ho, Jain, Abbeel NIPS 2006.11239 Probabilistic diffusion — basis of Stable Diffusion
2021 Score-Based SDEs Song et al. ICLR 2011.13456 Unified framework for score-based models

Era 7 — Transformer and LLMs (2017–2020)

Year Title Authors arXiv Contribution
2017 Attention Is All You Need Vaswani et al. NIPS 1706.03762 The Transformer — basis of every modern LLM
2018 BERT Devlin et al. NAACL 1810.04805 Bidirectional pre-training; SOTA on 11 NLU tasks
2018 GPT-1 Radford et al. OpenAI — Generative pre-training — transfer learning in NLP
2019 GPT-2 Radford et al. OpenAI — 1.5B params; zero-shot multitask
2020 GPT-3 Brown et al. NIPS 2005.14165 175B; in-context learning without gradient updates
2020 Scaling Laws Kaplan et al. OpenAI 2001.08361 Power-laws between scale, compute and data
2022 Chinchilla Hoffmann et al. DeepMind 2203.15556 Compute optimal: equal scaling of model and data

Era 8 — Vision with Transformers (2020–2021)

Year Title Authors arXiv Contribution
2020 ViT Dosovitskiy et al. ICLR 2010.11929 Pure Transformer for vision; image patches
2021 Swin Transformer Liu et al. ICCV 2103.14030 Hierarchical ViT with shifted windows; SOTA detection
2021 CLIP Radford et al. ICML — Contrastive image-text alignment; zero-shot

Era 9 — Reinforcement Learning (1988–2017)

Year Title Authors Venue Contribution
1988 TD Learning Sutton Machine Learning Temporal difference — basis of modern RL
1992 Q-Learning Watkins & Dayan Machine Learning Off-policy TD; foundation of DQN
2013 DQN Mnih et al. NIPS Deep Q-Networks; Atari at human performance
2015 DQN Nature Mnih et al. Nature Experience replay + target networks
2016 AlphaGo Silver et al. Nature MCTS + neural networks; defeats world champion
2017 AlphaZero Silver et al. Science Self-play RL; SOTA in chess, shogi, go
2017 PPO Schulman et al. ICLR 1707.06347 Stable policy gradient; standard in modern RL

Era 10 — Alignment and Post-Training (2022–2023)

Year Title Authors arXiv Contribution
2022 Chain-of-Thought Prompting Wei et al. NIPS 2201.11903 CoT dramatically improves reasoning
2022 Zero-Shot Reasoners Kojima et al. NIPS 2205.11916 "Let's think step by step" — zero-shot reasoning
2022 RLHF (InstructGPT) Ouyang et al. NIPS 2203.02155 Training with human feedback; basis of ChatGPT
2022 Constitutional AI Bai et al. Anthropic 2212.08073 AI feedback replaces human labels in alignment
2023 DPO Rafailov et al. NIPS 2305.18290 Direct Preference Optimization — no reward model
2023 ReAct Yao et al. ICLR 2210.03629 Reasoning + action in LLM agents
2023 Let's Verify Step by Step Lightman et al. ICLR 2305.20050 Process Reward Models for mathematics

Era 11 — Reasoning and Open-Source Frontier (2024–2026)

Year Title Authors arXiv Contribution
2024 Qwen2.5-Coder Hui et al. Alibaba 2409.12186 Complete recipe for code models
2024 SWE-bench Jimenez et al. ICLR 2310.06770 Benchmark of real GitHub issues
2024 FIM Bavarian et al. OpenAI 2207.14255 Fill-in-the-Middle — training for code completion
2024 DeepSeekMath/GRPO DeepSeek DeepSeek 2402.03300 GRPO — RL without a critic model
2025 DeepSeek-R1 DeepSeek DeepSeek 2501.12948 Pure RLVR; reasoning rivaling o1
2025 TurboQuant Zandieh et al. ICLR 2026 2504.19874 6× KV cache compaction, 8× speedup on H100
2025 EAGLE-3 SafeAILab NeurIPS 2503.01840 Speculative decoding 2–6× faster
2026 DeepSeek-V4 DeepSeek Technical report Apr/2026 — 1M context; CSA reduces KV cache 10×; 80.6% SWE-bench; MIT
2026 From AGI to ASI Legg, Hutter et al. (DeepMind) 2606.12683 Defines AGIASIAIXI; 4 paths to superintelligence + 6 frictions + physical limits. See agi-asi-superintelligence.md

Compression and Efficiency Papers

Year Title arXiv Contribution
2015 Deep Compression Han et al. ICLR Pruning + quantization + Huffman; 35-49× lossless
2015 Knowledge Distillation Hinton et al. NIPS Workshop Teacher-student; compresses large models
2022 GPTQ Frantar et al. ICLR 2210.17323 3-4 bit post-training quantization for LLMs
2023 AWQ Lin et al. MLSys 2024 2306.00978 Activation-aware weight quantization
2023 QLoRA Dettmers et al. NIPS 2305.14314 Fine-tuning with 4-bit; 70B on an RTX 3090
2024 BitNet b1.58 Ma et al. — 2402.17764 Ternary weights {-1,0,1}; 2.71× faster