MLOps for Large Models

models/
── kode-coder/
│   ├── v0.1.0/              ← Base version (Qwen2.5-Coder-32B)
│   │   ├── model/           ← Weights
│   │   ├── config.json
│   │   ├── tokenizer/
│   │   ├── eval/            ← Evaluation results
│   │   │   ├── swe-bench.json
│   │   │   ├── human-eval.json
│   │   │   └── privado.json
│   │   └── metadata.json    ← Date, commit, notes
│   │
│   ├── v0.2.0/              ← SFT fine-tune
│   │   ├── base: v0.1.0     ← Reference to the base model
│   │   ├── adapter/         ← LoRA weights
│   │   ├── config.json
│   │   ├── eval/
│   │   └── metadata.json
│   │
│   ├── v0.3.0/              ← SFT + DPO fine-tune
│   │   ├── base: v0.2.0
│   │   ├── adapter/
│   │   ├── config.json
│   │   ├── eval/
│   │   ── metadata.json
│   │
│   ── current → v0.3.0     ← Symlink to the active version

User → Load Balancer → Model v1 (vLLM)

                    ┌── Model v1 (blue) ──→ Production
User → LB ──────────┤
                    └── Model v2 (green) ──→ Staging

                    ┌── Model v1 ─→ 90% of traffic
User → LB ───────
                    └── Model v2 ──→ 10% of traffic

                    ┌── Model v1 ──→ Responds to the user
User → LB ──────┤
                    └── Model v2 ──→ Runs in shadow (no response)

# Deploy of model v1
vllm serve koder/kode-coder-v0.3.0 \
    --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 8192

# Deploy of model v2 (canary)
vllm serve koder/kode-coder-v0.4.0 \
    --port 8001 \
    --tensor-parallel-size 2 \
    --max-model-len 8192

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kode-model-v0-3-0
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kode-model
      version: v0.3.0
  template:
    metadata:
      labels:
        app: kode-model
        version: v0.3.0
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - koder/kode-coder-v0.3.0
        - --tensor-parallel-size
        - "2"
        resources:
          limits:
            nvidia.com/gpu: 2
---
apiVersion: v1
kind: Service
metadata:
  name: kode-model
spec:
  selector:
    app: kode-model
    # version: v0.3.0  ← change to v0.4.0 to switch version
  ports:
  - port: 80
    targetPort: 8000

Production → Metrics collection (LangFuse, Prometheus) → Dashboard (Grafana)
                                        ↓
                               Alerts (Slack, email)
                                        ↓
                               Analysis (weekly)
                                        ↓
                               Decision: keep, rollback, retrain

def detect_drift(current_eval, baseline_eval, threshold=0.05):
    """Detect drift by comparing current eval with baseline."""
    metrics = ["code_pass_rate", "llm_judge_score", "latency_p95"]

    drifts = []
    for metric in metrics:
        current = current_eval[metric]
        baseline = baseline_eval[metric]
        change = abs(current - baseline) / baseline

        if change > threshold:
            drifts.append({
                "metric": metric,
                "change": change,
                "current": current,
                "baseline": baseline,
            })

    return drifts

# Usage
drifts = detect_drift(eval_semanal, eval_baseline)
if drifts:
    alert(f"Drift detected: {drifts}")

# Rate limiting with Redis
import redis
import time

class RateLimiter:
    def __init__(self, redis_client, max_requests=100, window=60):
        self.redis = redis_client
        self.max_requests = max_requests
        self.window = window

    def is_allowed(self, user_id):
        key = f"rate_limit:{user_id}"
        current = self.redis.get(key)

        if current is None:
            self.redis.setex(key, self.window, 1)
            return True

        if int(current) >= self.max_requests:
            return False

        self.redis.incr(key)
        return True

# Token quota
class TokenQuota:
    def __init__(self, redis_client, daily_limit=100000):
        self.redis = redis_client
        self.daily_limit = daily_limit

    def check_and_consume(self, user_id, tokens):
        key = f"token_quota:{user_id}:{date.today()}"
        used = int(self.redis.get(key) or 0)

        if used + tokens > self.daily_limit:
            return False

        self.redis.incrby(key, tokens)
        return True

# 1. Switch to the previous version (Kubernetes)
kubectl set image deployment/kode-model \
    vllm=vllm/vllm-openai:latest \
    --record

# 2. Or change the symlink (file-based)
ln -sfn models/kode-coder/v0.3.0 models/kode-coder/current

# 3. Or swap the weight in vLLM (hot reload)
curl -X POST http://localhost:8000/v1/reload \
    -d '{"model": "koder/kode-coder-v0.3.0"}'

# 4. Check health
curl http://localhost:8000/health

# 5. Notify the team

                   ┌── Pod 1 (vLLM) ──→ GPU 1
Load Balancer ─────┼── Pod 2 (vLLM) ──→ GPU 2
                   ┼── Pod 3 (vLLM) ──→ GPU 3
                   └── Pod N (vLLM) ──→ GPU N

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kode-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kode-model
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: gpu_utilization
      target:
        type: AverageValue
        averageValue: 80

Cost per request = (GPU-hours per request) × (cost per GPU-hour)

GPU-hours per request = (tokens generated × time per token) / 3600

Example:
  - 500 tokens generated
  - 50ms per token = 25s total
  - 25s / 3600 = 0.007 GPU-hours
  - A100 cost: $2.80/hour
  - Cost per request: 0.007 × $2.80 = $0.02

Versioning: W&B Artifacts + HuggingFace Hub (private)
Deployment: Kubernetes + vLLM
Monitoring: Prometheus + Grafana + LangFuse
Alerts: Slack + PagerDuty
Rate limiting: Redis + NGINX
CI/CD: GitHub Actions → build Docker → push → deploy

1. New checkpoint → Automatic eval (SWE-bench, private eval)
2. If eval OK → Push to HuggingFace Hub + W&B
3. Canary deploy (10% of traffic)
4. Monitor 24h
5. If OK → Gradual ramp-up (25% → 50% → 100%)
6. If a problem → Automatic rollback