Vol. VIII · Production Systems · VIII.3
Observability
A trace is the loop laid out in time: prompts, tool I/O, token counts, costs, user id. Without it you cannot eval, debug, or bill.
5 min read
Doctrine
- Trace the constructed window, not only the user utterance.
- OpenTelemetry where you can. Vendor lock on traces is how you lose forensics.
- Cost per successful task is the metric finance will ask for. Have it.
Default plane
Langfuse is the open default. LangSmith is the LangChain-native one. OpenAI Agents SDK ships tracing. Whatever you pick, the schema is the same: spans around llm and tool, attributes for tokens and model, links to eval runs.
Anti-patterns
- Logging completions but not tool results.
- PII in traces with no redaction policy.