Playbook
Distill a trace
Compile a paid-for lesson into a skill, a rubric, or a golden case. Do not hoard transcripts.
When: The same failure happened twice, or a run was surprisingly good.
- 01
Open the trace
Langfuse (or equivalent). Name the missing rule, the bad schema, or the missing eval.
Observability - 02
Write the artifact
Skill, promptfoo case, or DSPy metric. Smallest artifact that would have prevented the miss.
Distill Traces into Skills - 03
Do not store the novel
Memory writes are curated tool calls. Session logs are not memory.
Long-Term Memory
Done when
- A new eval case and/or a SKILL.md, linked from the trace. Raw transcript is not 'memory'.
Refuse
- Memory as unfiltered logs. Memory writes are tool calls. Curate. Letta's core vs archival. User can inspect/delete.
- Eval by demo. 20–50 real tasks before the framework. promptfoo, Inspect, DSPy. CI on prompt change.