Skill · meta · distill-traces
Distill a trace
Compile a paid-for agent run into a skill, a rubric, or a golden eval. Use when the same miss happened twice, a run was surprisingly good, the user says distill, 'learn from this', 'turn this into a skill', or when memory is being implemented as unfiltered logs.
GET /api/canon/skills/distill-traces?format=md
Session transcripts are not memory. Memory writes are curated tool calls. The compounding loop of a good harness is trace → skill or eval → CI.
When
You have a trace. The lesson is sitting in tokens you already paid for.
Do
- 01
Open the trace
Name the missing rule, the bad schema, or the missing eval. One miss, one artifact.
- 02
Write the smallest artifact
A SKILL.md, a promptfoo case, or a DSPy metric that would have prevented the miss. Prefer an eval if you can score it; a skill if it is procedure.
- 03
Do not store the novel
Curate. Letta-style core vs archival. The user can inspect and delete. Secrets never enter memory.
Don't
- Stuff the transcript into the next session.
- Write a 2,000-line skill that restates the log.
- Skip the eval because 'we'll remember next time'.
Hard rules
- Same miss twice → eval case is mandatory.
- Surprisingly good run → skill or rubric.
- Raw logs are observability, not memory.
Refuse
- Memory as unfiltered logs — Session transcripts stuffed into the next session. The user cannot inspect or delete them.
- Eval by demo — Architecture first, golden set never. Success is a recorded GIF.
Load with this
Load next
Trigger tests
Should fire
- “Turn this failed run into a skill”
- “We keep making the same refund mistake”
- “Distill yesterday's trace”
Should not
- “Add logging to the API”
- “Set up Sentry for the web app”