Continuous Model Evaluation

Private eval loop, calibrated LLM-as-a-judge, canary strings, contamination and drift detection. Updated in April 2026.

Overview

Public benchmarks (MMLU, HumanEval, SWE-bench) measure generic performance. But for a proprietary model, what matters is: is it improving at what matters to us?

Continuous evaluation answers:

  1. Did the model improve after the last fine-tune?
  2. Did it regress on something that used to work?
  3. Is there contamination of the training data with the evaluation data?
  4. Is quality degrading in production?

Components of the Eval Loop

New checkpoint → Automatic eval → Dashboard → Decision (merge/reject/rollback)
                         ↓
                  Comparison with baseline
                         ↓
                  Regression? → Alert

1. Private Evaluation Dataset

Principles

Principle Why
Never published If the eval was published, the model may have been trained on it
Diversified Cover all of the product's use cases
Up to date Include recent examples (last 3–6 months)
Versioned Each eval version is immutable — compare only the same version
Segmented Separate by domain (code, text, reasoning, etc.)

Building the private eval

Phase 1: Collect real use cases
  → Resolved support tickets
  → Examples of successful prompts
  → Edge cases the current model does not solve

Phase 2: Create reference input/output pairs
  → For each prompt, create the expected output (gold)
  → Have 2–3 reference outputs per prompt (accepted variations)

Phase 3: Categorization
  → Code (unit tests, refactoring, debugging)
  → Text (summarization, translation, generation)
  → Reasoning (mathematics, logic, planning)
  → Multimodal (if applicable)

Phase 4: Size
  → Minimum: 500 examples per category
  → Ideal: 2000–5000 per category
  → Total: 2K–20K examples

Example structure

eval_v1/
├── coding/
│   ├── unit_tests/         # 500 examples
│   ├── refactoring/        # 500 examples
│   ├── debugging/          # 500 examples
│   └── code_review/        # 500 examples
├── text/
│   ├── summarization/      # 300 examples
│   ├── translation/        # 300 examples
│   ── generation/         # 300 examples
├── reasoning/
│   ├── math/               # 300 examples
│   ├── logic/              # 300 examples
│   └── planning/           # 300 examples
└── metadata.json           # version, date, description

2. Evaluation Metrics

Automatic metrics

Metric When to use Tool
Exact Match Output must be identical == string comparison
Token overlap (F1) Output similar but not identical rouge_score
Code execution Does the generated code work? Execute + verify output
Unit test pass rate Does the code pass the tests? pytest / unittest
Semantic similarity Is the meaning preserved? Embeddings + cosine
Format compliance Output in the right format? Regex / parser

Code evaluation (the most important one for Kode)

def evaluate_code_generation(model_output, test_cases):
    """Evaluate generated code by running tests."""
    results = []
    for test in test_cases:
        try:
            # Execute the generated code
            exec(model_output, test.globals)
            # Run the test
            result = test.run()
            results.append(result.passed)
        except Exception as e:
            results.append(False)
    return sum(results) / len(results)  # pass rate

Main metric: % of unit tests passed — it is the most objective metric and the one most correlated with real code quality.

LLM-as-a-Judge

When output is not automatically verifiable (text, reasoning):

Prompt: "Rate the quality of the answer below from 1 to 10.
Criteria: correctness, completeness, clarity.

Model answer: {model_output}
Reference answer: {gold_output}

Score:"

Problem: LLM-as-a-judge has bias (it prefers long, verbose answers).

Solution: Calibration

  1. Create a dataset of 200–500 examples with human scores
  2. Measure the correlation of the LLM-as-a-judge with the human scores
  3. If correlation < 0.7, adjust the prompt or change the judge model
  4. Re-calibrate every 3 months
Category Main metric Secondary metric Minimum threshold
Code generation Unit test pass rate Token overlap with reference > 80%
Refactoring Diff correctness (human eval) Syntax validity > 90% syntax ok
Debugging Bug fixed rate Explanation quality (LLM judge) > 70%
General text Semantic similarity Format compliance > 0.8 cosine
Reasoning Correctness rate Step-by-step accuracy > 75%

3. Canary Strings

What they are

Canary strings are unique texts injected into the training data to detect contamination:

# Canary string in the training dataset:
"The canary code canary-7f3a9b2c is used to detect contamination."

# In the eval, check whether the model "remembers" the canary:
prompt = "What is the canary string of the training dataset?"
# If the model answers correctly → contamination detected!

How to use

  1. Insert canaries into the training data (1 canary per 1M tokens)
  2. Include questions about canaries in the eval
  3. If the model gets the canaries right → training-to-eval leakage

Canaries for public datasets

Many public benchmarks (MMLU, HumanEval) use canaries to detect whether models were trained on the evaluation data:

Benchmark Canary
MMLU Specific texts injected into subsets
HumanEval Unique comments in the docstrings
GSM8K Problems with specific numbers

For Kode: Always check whether the benchmarks we use have canaries and respect the evaluation protocol.

4. Contamination Detection

Types of contamination

Type Description Detection
Train → Eval Eval data leaked into training Canary strings, perplexity check
Train → Public benchmark Public benchmark in training Check the model's papers/datasets
Eval → Train Eval generated by the model goes into training Rigorous versioning

Perplexity check

If the model has suspiciously low perplexity on the eval, there may be contamination:

def check_contamination(model, eval_dataset):
    """Check whether perplexity on the eval is abnormally low."""
    ppl_eval = model.perplexity(eval_dataset)
    ppl_baseline = model.perplexity(baseline_dataset)

    # If perplexity on the eval is < 50% of the baseline → suspected contamination
    if ppl_eval < 0.5 * ppl_baseline:
        return "ALERT: Possible contamination detected"
    return "OK"

5. A/B Testing of Models

When to use

  • Compare two model versions before deploy
  • Test fine-tune vs base model
  • Validate that the new version did not regress

Methodology

1. Select 200–500 representative prompts
2. Generate output with Model A and Model B
3. Evaluate with automatic metrics + LLM-as-a-judge
4. Compare statistically (t-test or Mann-Whitney)
5. Decide: A better, B better, or tie

Comparison metrics

Metric Model A Model B Difference
Code pass rate 82% 87% +5% ✓
Latency (ms/token) 120 140 +17% ✗
Token overlap 0.78 0.81 +4% ✓
LLM judge score 7.2 7.8 +8% ✓
Verdict — B wins —

6. Production Monitoring

Metrics to monitor

Metric Alert if Action
Quality (eval score) Drops > 5% vs baseline Investigate, possibly rollback
Latency p95 Increases > 20% Optimize inference, scale
Error rate Increases > 1% Check logs, debug
Token usage Changes drastically Check for prompt injection or loop
User feedback Rating drops Re-train with negative feedback

Drift detection

Models can degrade in production due to:

  • Data drift: production data changes (e.g., new code libraries)
  • Concept drift: what was "good" changes (e.g., new coding practices)
  • Prompt drift: users change how they interact

Detection:

1. Keep a buffer of production prompts (last 7 days)
2. Run automatic eval on this buffer weekly
3. Compare with baseline (previous month)
4. If difference > threshold → drift alert

Canary deployment

New model → 5% of traffic → Monitor 24h → If OK → 25% → 50% → 100%
                                    ↓
                              If problem → Immediate rollback

Tools

lm-evaluation-harness (EleutherAI)

Standard framework for LLM evaluation:

lm_eval --model hf \
    --model_args pretrained=my-model \
    --tasks hellaswag,arc_easy,human_eval \
    --batch_size auto

Advantages: 100+ benchmarks, open-source, active community.

SWE-bench runner

For code evaluation:

# Run SWE-bench on the model
python run_evaluation.py \
    --model my-model \
    --dataset swe-bench-verified \
    --num-workers 32

W&B Sweeps + Eval

import wandb

wandb.init(project="kode-eval")
wandb.log({
    "code_pass_rate": 0.87,
    "latency_p95": 140,
    "llm_judge_score": 7.8,
    "regression_detected": False,
})

LangFuse

For eval in production:

  • Trace of each request
  • Automatic evaluation with LLM-as-a-judge
  • Quality dashboard over time
  • Integrated user feedback

For Kode

Phase 1: Initial setup (1–2 weeks)
  → Create a private dataset of 5K examples (code, text, reasoning)
  → Configure lm-eval-harness with public benchmarks
  → Configure SWE-bench runner
  → Set up W&B for tracking

Phase 2: Automatic evaluation (continuous)
  → Every new checkpoint runs the full eval
  → Dashboard with comparison vs baseline
  → Automatic alert if regression > 3%

Phase 3: Production monitoring (continuous)
  → LangFuse for request tracing
  → Canary deployment for new models
  → Feedback loop with users

Phase 4: LLM-as-a-judge calibration (monthly)
  → Collect 200 examples with human scores
  → Check correlation with the LLM judge
  → Adjust the prompt if needed

Minimum private dataset for Kode

Category Examples Source
Unit tests 500 Tests from Koder projects
Refactoring 300 Refactoring PRs
Debugging 300 Resolved issues
Code review 200 Review comments
Code generation 500 Specifications → code
Text (docs) 300 Technical documentation
Reasoning 200 Design problems
Total 2,300 —

References

Resource Description
lm-evaluation-harness EleutherAI's eval framework
SWE-bench Coding benchmark on real repos
W&B Sweeps Experiment tracking
LangFuse LLM observability and eval
Canaries in ML Papers on canary strings for contamination detection