Vol. X · Field Operations · X.6
Distill Traces into Skills
A trace is a lesson the agent already paid for. Compiling it into a skill, a rubric, or a golden eval is how the system gets cheaper and less lost. Uncompiled traces are landfill.
6 min read
langfuse/langfuseanthropics/skillspromptfoo/promptfoostanfordnlp/dspymuratcankoylan/Agent-Skills-for-Context-Engineering
Doctrine
- When a run fails twice the same way, write a skill or an eval case, not a longer system prompt.
- Memory writes are curated. Session logs are not memory.
- DSPy-style compilation and skill authoring are the same job at different altitudes: turn traces into policy.
The compile step
Langfuse gives you the trace. A human (or a meta-agent with a rubric) extracts: the missing rule, the bad tool schema, the eval case. That becomes a SKILL.md, a promptfoo test, or a DSPy metric. This is Factor 2 over time — you own the prompt because you keep rewriting it from evidence.
Anti-patterns
- Storing raw transcripts as 'memory' and stuffing them into the next session.
- Never looking at traces except when a customer yells.