# LLM Evaluation and Observability Stack

> A technical stack for comparing models, tracing AI behavior, measuring quality and cost, and controlling releases with evidence.

- Canonical: https://gptnavi.com/stacks/llm-evaluation-and-observability-stack
- Estimated monthly cost: $20-$500/month plus model usage
- Last materially updated: 2026-08-19
- Best for: AI product teams, Developers, Technical founders, Platform teams

## Problems this stack solves

- Model selection
- Prompt regressions
- Hidden AI cost
- Low-quality outputs
- Unobservable agent behavior

## Recommended tools

### Model access

Recommended: OpenRouter, ngrok AI Gateway

Why: Test model alternatives through a consistent access layer with routing and cost visibility.

### Evaluations

Recommended: LangSmith, DeepSeek

Why: Build fixed task sets, review edge cases, and use a second model only as a supporting evaluator.

### Observability

Recommended: Langfuse, PostHog

Why: Trace prompts, tool calls, latency, corrections, product behavior, and spend.

### Implementation hardening

Recommended: Codex, Windsurf

Why: Add input validation, approval gates, retries, safe fallbacks, and regression tests.

## Beginner setup plan

1. Choose one customer-facing task and collect representative test cases.
2. Compare quality, safety, latency, and cost before choosing a default model.
3. Trace every production workflow and sample corrections.
4. Block release when a fixed regression test fails.

## Included workflows

- https://gptnavi.com/workflows/llm-evaluation-before-production
- https://gptnavi.com/workflows/ai-product-release-control-plane
- https://gptnavi.com/workflows/api-to-ai-operations-workflow
- https://gptnavi.com/workflows/ai-agent-ops-from-recurring-task