Vol. VIII · Production Systems · VIII.2
Evaluation
Evals are tests. promptfoo, DSPy, Langfuse datasets, and the Hugging Face agents course all assume you have a golden set before you have an architecture.
7 min read
Doctrine
- Write 20–50 real tasks before picking a framework.
- Score trajectories, not just final answers: did it call the dangerous tool?
- Keep a regression suite. Prompt edits without evals are untested deploys.
The eval stack
promptfoo: declarative cases, CI, red team. DSPy: optimize against the metric. Langfuse: production traces become datasets. Google ADK ships evaluation as a first-class kit concern. Hugging Face's course forces you to measure smolagents, which is the point.
Multi-agent systems need extra metrics: extra tokens, extra failures, extra latency versus a single agent. If the crew does not win on the golden set, delete the crew.
Anti-patterns
- Demo-driven development: one happy path on stage.
- LLM-as-judge with no correlation study against humans.