Observability and Monitoring for LLMs

Why Observability in LLMs?

• Debug: LLMs are black boxes — see what goes in and out of each pipeline stage
• Costs: Monitor spend per API (input/output tokens per endpoint)
• Quality: Track latencies, error rate, response quality
• Feedback: Collect thumbs-up/down from users for preference data

LLM Observability Platforms

LangFuse

• URL: langfuse.com
• Open-source: Yes (self-hostable)
• Features: Traces, spans, versioned prompts, scores, datasets, playground
• Integration: langfuse SDK for Python/JS; integrations with LangChain, LlamaIndex, OpenAI
• When to use: Self-hosted; open-source; privacy-first
• Self-host: Docker compose; PostgreSQL

from langfuse.openai import openai  # drop-in replacement
response = openai.chat.completions.create(...)  # auto-traced

Phoenix (Arize)

• URL: phoenix.arize.com · github.comArize-aiphoenix
• Open-source: Yes
• Features: Traces (OpenTelemetry), automatic evals, datasets, embeddings explorer
• Integration: OpenTelemetry-native; works with any LLM
• Differentiator: Built-in evals (hallucination, relevance, toxicity)

Helicone

• URL: helicone.ai
• Model: Reverse proxy — swap openai.com for oai.helicone.ai
• Features: Costs, latencies, rate limiting, caching, user tracking
• Zero-code: Just change the base URL
• When to use: Monitoring API costs without code

Braintrust

• URL: braintrustdata.com
• Focus: Integrated evals + datasets + tracing
• Differentiator: CI/CD for evals — runs a benchmark on every PR

PromptLayer

• URL: promptlayer.com
• Focus: Prompt versioning + tracing
• When to use: Teams with many prompts in production

Weights & Biases (Wandb)

• URL: wandb.ai
• Primary focus: Model training (loss curves, hyperparameters, artifacts)
• LLM features: Prompt versioning, traces, evals, Weave framework
• When to use: Already use wandb for training; also want to monitor production

MLflow

• URL: mlflow.org (Linux Foundation)
• Open-source: Yes (self-hostable)
• Features: Experiment tracking, model registry, serving, tracing (LLM)
• Backend: PostgreSQL + S3/MinIO for artifacts
• When to use: Fully open-source stack; integration with Spark/Databricks

Data Pipeline

datatrove (HuggingFace)

• URL: github.comhuggingfacedatatrove
• Focus: Large-scale pre-training data processing
• Features: Readers (Common Crawl, Parquet, JSON), filters, deduplication, writers
• Parallelism: Native; processes petabytes of data
• Use: FineWeb was built with datatrove

HuggingFace Datasets

• URL: huggingface.codocsdatasets
• Features: Arrow format, streaming, parallel map/filter, pushtohub
• Integration: PyTorch DataLoader, JAX, Spark
• Deduplication: MinHash LSH via datasets.dedup

datasketch

• URL: github.comekzhudatasketch
• Algorithms: MinHash LSH, HyperLogLog, TopK
• Use: Near-duplicate deduplication in text/code datasets

Profiling and Performance

PyTorch Profiler

with torch.profiler.profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    model(x)
print(prof.key_averages().table(sort_by="cuda_time_total"))

• Identifies bottlenecks: which op consumes the most time/memory

NVIDIA Nsight Systems / Nsight Compute

• Nsight Systems: CPU+GPU timeline; visualization of streams and gaps
• Nsight Compute: Per-kernel profiling; roofline analysis
• Use: Identify whether we are memory-bound or compute-bound

torch.cuda.memory_summary()

print(torch.cuda.memory_summary(device=None, abbreviated=False))

• Tracks where VRAM is being consumed

Production Metrics

Latency

Throughput

Cost

OpenTelemetry for LLMs

• Emerging standard: OpenTelemetry Semantic Conventions for LLMs (OTEL SIG)
• Attributes: gen_ai.system, gen_ai.model, gen_ai.input.tokens, gen_ai.output.tokens
• Exporters: Jaeger, Zipkin, Prometheus, OTLP (for Phoenix, Grafana, etc.)

Recommended Observability Stack for Kode

Development:        LangFuse self-hosted (traces + versioned prompts)
Training:           W&B (loss curves, artifacts, evals)
Production:         LangFuse (traces) + Prometheus/Grafana (metrics)
Costs:              Helicone (if using OpenAI/Anthropic API)
Continuous evals:   Phoenix (RAGAS score, hallucination detection)