Data Pipeline for LLMs

Data curation pipeline at scale: crawler, deduplication, quality filtering, PII scrubbing. Updated April 2026.

Overview

Training data quality is the #1 factor in an LLM's performance. A model trained on 1T well-curated tokens beats a model trained on 5T dirty tokens.

The typical pipeline has 5 stages:

Collection → Deduplication → Filtering → PII Scrubbing → Final Formatting

Each stage reduces the volume but dramatically increases quality.

Stage 1: Collection (Crawler)

Data sources

CommonCrawl

CommonCrawl is the primary source of web data. Every month (~1B pages, ~5PB raw).

Typical pipeline:

WARC files (raw) → WET files (extracted text) → Filtering → Dedup → Clean

Tools:

• cc_net (Facebook) — complete CC-to-training pipeline
• datatrove (HuggingFace) — modular pipeline in Python
• resiliparse — fast text extraction from HTML (Rust)
• trafilatura — main-content extraction from web pages

For code

For Kode: The Stack v2 is the best code source — already filtered, deduplicated, with license metadata.

Stage 2: Deduplication

Deduplication is the most underrated and most important stage. Duplicate data causes:

• Overfitting — the model memorizes instead of generalizing
• Bias — repeated texts weigh more in the gradient
• Inefficiency — compute wasted processing the same information

Deduplication levels

MinHash + LSH (Near-Deduplication)

The industry's gold-standard technique (used by RefinedWeb, FineWeb, RedPajama):

1. Shingling: split document into shingles of n-grams (e.g., 5-grams of words)
2. MinHash: apply ~200 hash functions, take the minimum of each → 200-int signature
3. LSH (Locality Sensitive Hashing): group similar signatures into buckets
4. Compare within buckets: Jaccard similarity > 0.8 → duplicate

Performance: Processes 100B documents in ~24 hours with 64 cores.

Tools:

• datasketch (Python) — MinHash + LSH implementation
• text-dedup (HuggingFace) — complete deduplication pipeline
• Fineweb deduplication — reference implementation

Typical results

Rule of thumb: expect to remove 30–40% of the dataset from deduplication alone.

Stage 3: Quality Filtering

After deduplication, filter what remains to remove junk:

Heuristic filters

Model-based filters

RefinedWeb (Falcon) filtering pipeline

Industry reference:

1. URL filter: remove low-quality domains
2. Heuristic filter: length, repetition, boilerplate
3. Language filter: fastText (keep only English, Spanish, etc.)
4. Quality filter: classifier trained on Wikipedia vs spam
5. PII filter: regex + NER
6. Deduplication: MinHash + LSH at the end

Result: From 5PB of raw CommonCrawl → 1.8T high-quality tokens.

Stage 4: PII Scrubbing

PII (Personally Identifiable Information) includes:

• Full names
• Email addresses
• Phone numbers
• CPFs, RGs, documents
• Physical addresses
• Credit card numbers
• IPs

Techniques

Microsoft Presidio

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

text = "Meu email é rodrigo@koder.dev e meu CPF é 123.456.789-00"
results = analyzer.analyze(text=text, entities=["EMAIL_ADDRESS", "BR_CPF"], language='pt')
anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
# "Meu email é <EMAIL_ADDRESS> e meu CPF é <BR_CPF>"

For Kode: Use Presidio with custom entities for Brazil (CPF, CNPJ, RG, BR phone).

Quality loss from PII scrubbing

Aggressive scrubbing can remove valuable context. Example:

• "rodrigo@koder.dev" → "<EMAIL>" (ok, we lose nothing)
• "John Smith wrote a paper" → "<PERSON> wrote a paper" (we lose the name, but the sentence still makes sense)

Recommended strategy:

1. Redact sensitive PII (emails, CPFs, phones) → replace with placeholder
2. For proper names in generic context → keep (it is not PII if it does not identify a specific person)
3. Document what was scrubbed for auditing

Stage 5: Final Formatting

Tokenization and packing

Clean texts → Tokenization (BPE/SentencePiece) → Token sequences
                                                   ↓
                                    Packing
                                                   ↓
                                    .bin or .parquet files

Packing vs. Padding:

• Packing: Concatenate short documents until the context window is full (no waste)
• Padding: Fill short documents with padding tokens (compute waste)

Packing is preferable — up to 30% more compute-efficient.

Storage formats

Tools and Frameworks

datatrove (HuggingFace)

Modular pipeline for data processing at scale:

from datatrove.pipeline import (
    DocumentTokenizer,
    MinhashDedupSignature,
    MinhashDedupCluster,
    QualityFilter,
    PIIExtraction,
)

pipeline = [
    DocumentTokenizer(),
    MinhashDedupSignature(num_permutations=200),
    MinhashDedupCluster(threshold=0.8),
    QualityFilter(min_length=50, max_length=100000),
    PIIExtraction(),
]

Advantages: Modular, scalable (Spark/Ray), open-source.

cc_net (Facebook)

Complete CommonCrawl → training-dataset pipeline:

• Text extraction from WARC/WET
• Filtering by language, quality, length
• Deduplication
• Tokenization

RedPajama Data Pipeline

Open source of the pipeline that produced RedPajama v2 (30T tokens):

• GitHub: togethercomputer/RedPajama-Data
• Includes crawling, filtering, dedup scripts

Spark + Ray

For massive scale (10T+ tokens):

• Apache Spark — distributed batch processing
• Ray Data — modern pipeline, easier than Spark

Dataset Quality Metrics

For Kode

Recommended pipeline

To build Kode's training dataset:

Phase 1: Collection (1–2 weeks)
  → The Stack v2 (code) — already curated
  → CommonCrawl + cc_net (general text)
  → Technical documentation (lib, framework docs)

Phase 2: Deduplication (3–5 days)
  → MinHash + LSH (threshold 0.8)
  → Remove cross-source near-duplicates

Phase 3: Filtering (2–3 days)
  → Heuristics: length, repetition, boilerplate
  → Language ID: keep Portuguese + English
  → Quality classifier: filter spam, junk

Phase 4: PII Scrubbing (1 day)
  → Presidio with custom BR entities
  → Redact emails, CPFs, phones

Phase 5: Tokenization and Packing (2–3 days)
  → Qwen2.5 or Llama 3 tokenizer
  → Packing with 8K–32K context window
  → Format: Parquet for storage, MDS for training

Volume estimate

Hardware for the pipeline

Total estimated cost: R$ 5–15K (mostly broadband and storage).

Papers and References