Training Datasets

Pretraining Datasets — General Text

Dataset Size License Source Strengths
FineWeb 15T tokens Open 96 Common Crawl snapshots Best general open-source dataset; beats C4, Pile, RedPajama
FineWeb-Edu 1.3T tokens Open Filtered FineWeb Best for MMLU/ARC; high-quality educational content
RedPajama-V2 30T tokens Mixed Common Crawl Largest open dataset; variable quality
RedPajama-V1 1.2T tokens Mixed CC + GitHub + arXiv + Wikipedia Reproducible; composition: 878B CC, 59B GitHub, 28B arXiv
SlimPajama 627B tokens Open Deduplicated RedPajama-V1 49.6% smaller than original; better quality per token
Dolma 3T tokens Open Varied Accessible for data research
The Pile 886GB (570B tokens) Open 22 sub-datasets EleutherAI; historical baseline; thematic diversity
ROOTS 1.6TB Mixed 498 datasets 59 languages; multilingual; community-curated (BLOOM)
C4 175B tokens CC-BY 1 Common Crawl snapshot Widely used; extensive filtering
DCLM 240T tokens pool Open Common Crawl Benchmark testbed for curation; 7B at 64% MMLU

Links:

  • FineWeb: huggingface.codatasetsHuggingFaceFW/fineweb
  • FineWeb-Edu: huggingface.codatasetsHuggingFaceFW/fineweb-edu
  • RedPajama: huggingface.codatasetstogethercomputer/RedPajama-Data-V2
  • Dolma: huggingface.codatasetsallenai/dolma

Code Datasets

Dataset Size Languages License Source Use
The Stack v2 67.5TB 604+ Permissive 104M GitHub repos (Software Heritage) Code pretraining
The Stack v1 3.1TB 30+ Permissive 137M GitHub repos (2015–2022) Code pretraining (older)
GitHub Archive Continuous Multi Varies GH events (commits, PRs) Metadata + diffs
CommitPack — Multi Varies diff + commit message Diff/review training
SWE-bench Train — Python Varies Real issues/PRs Agent training
APPS 5K problems Python Open Competitions Coding contest training
CodeContests — Multi Open Codeforces + AtCoder Competitive programming
LiveCodeBench Continuous Multi Open Recent problems Eval without contamination

Math Datasets

Dataset Size Type License Use
OpenMathInstruct-1 1.8M pairs Synthetic (Mixtral) Permissive SFT for math
OpenMathInstruct-2 Larger Synthetic (Llama 3) Permissive Expanded version
MetaMathQA — Bootstrap Open Variations of math problems
MATH dataset 12.5K problems Competitions Open Eval + fine-tuning (levels 1–5)
GSM8K 8.5K problems Grade school Open Fine-tuning basic math reasoning
NuminaMath 860K problems Competitions Open Training advanced math reasoning

Instruction and Chat Datasets

Dataset Size License Use
UltraChat 1.5M dialogues Open Multi-turn conversation training
WizardLM 250K instructions Open Complex instructions (Evol-Instruct)
Orca-Math 200K problems Open Math via Agent-Instruct
ShareGPT Varied Mixed Real shared ChatGPT conversations
OpenHermes 2.5 900K Open General instruction; high quality
Alpaca 52K Non-commercial Original Self-Instruct (GPT-3.5 generated)

Multilingual Datasets

Dataset Languages Tokens License
ROOTS 59 (46 nat. + 13 prog.) — Mixed
BLOOM 46 — BigScience RAIL
mC4 101 — CC-BY
CulturaX 167 6.3T Mixed
Qwen 3.5 training 201 — —

Multimodal Datasets

Dataset Type Size License Use
LAION-5B Image-text 5B pairs Open Vision model pretraining
CC3M/CC12M Image-text 3M/12M CC-BY Conceptual; image captions
WIT Image-text 37M Apache Wikipedia image-text
COCO Detection/segmentation 330K images CC-BY Classic vision benchmark

Evaluation Datasets (Not for Training)

Never include these datasets in training — contamination invalidates benchmarks.

  • MMLU: 15K multiple-choice questions (57 subjects)
  • HumanEval: 164 Python problems
  • SWE-bench Verified: 500 human-validated GitHub issues
  • MATH-500: Subset of 500 competition problems
  • AIME 2024/2025: 30 olympiad-level problems

Quality > Quantity

Finding Source
FineWeb-Edu (1.3T) > RedPajama (1.2T) on MMLU HuggingFace, 2024
Phi-1 (7B synthetic tokens) beats models trained on >1T tokens on code Microsoft Phi, 2023
SlimPajama (627B) > RedPajama (1.2T) — deduplication matters Together AI, 2023
Chinchilla: 20 tokens/parameter is compute-optimal DeepMind, 2022