# LLM evaluation before production

> Compare models and prompts against a fixed task set before an AI feature reaches customers, with traces, cost limits, and human release approval.

- Canonical: https://gptnavi.com/workflows/llm-evaluation-before-production
- Category: Coding
- Difficulty: Advanced
- Setup time: 4 hours
- Estimated time saved: 4-10 hours
- Last materially updated: 2026-08-19
- Best for: AI product teams, Developers, Technical founders, Platform teams
- Tools: OpenRouter, LangSmith, Langfuse, Codex, DeepSeek

## Quick answer

This workflow prevents model selection by demo alone. It evaluates a real task set for accuracy, safety, latency, cost, and operator correction before a production release.

## When to use it

- AI support drafts
- Document extraction
- Internal knowledge assistant
- Classification workflow

## Steps

1. **Build a representative test set** — Collect anonymized examples that cover routine, ambiguous, adversarial, missing-data, and edge-case requests with expected outcomes. Tool: LangSmith. Expected output: A versioned evaluation dataset.
2. **Run model comparisons** — Test approved models through a consistent interface and record output quality, latency, token use, and failure cases. Tool: OpenRouter. Expected output: Comparable model runs.
3. **Trace real workflow behavior** — Capture prompts, tool calls, latency, cost, and errors in a controlled pre-production environment. Tool: Langfuse. Expected output: A trace and observability baseline.
4. **Fix the workflow, not just the prompt** — Use a coding agent to implement input validation, source requirements, approval gates, retries, and safe fallbacks. Tool: Codex. Expected output: A hardened feature branch.
5. **Run final challenge cases** — Use a second model as one reviewer for missed risks, then have a human owner approve the release decision. Tool: DeepSeek. Expected output: A signed release checklist.

## Prompt templates

### Evaluation rubric

Create an evaluation rubric for this AI feature. Include accuracy, citation or evidence quality, safety, privacy, refusal behavior, latency, cost, correction burden, and escalation. Feature: [paste]

### Failure analysis

Analyze these AI failures. Group root causes into input quality, prompt design, retrieval, tool use, model limits, unsafe action, and UX ambiguity. Recommend the smallest reliable fix and a regression test. Runs: [paste]

## Common mistakes

- Choosing a model from one impressive demo
- Evaluating only happy paths
- Shipping without a human escalation path

## Related workflows

- https://gptnavi.com/workflows/ai-product-release-control-plane
- https://gptnavi.com/workflows/api-to-ai-operations-workflow
- https://gptnavi.com/workflows/ai-agent-ops-from-recurring-task