Skip to content
Canon

Playbook

Design an agent

Evals first. Raw loop second. Framework last, and only for a missing primitive.

When: A new agentic surface, before anyone opens a framework README.

  1. 01

    Write the golden set

    20–50 real tasks with expected structured outcomes. Include the failures you already know. If you cannot score it, you do not have a product yet.

    Evaluation
  2. 02

    Name the path

    If you can name the stages, you want a workflow (chain, route, parallel, eval-opt). If the environment talks back, you want an agent with a budget. Multi-agent last.

    Workflows versus Agents
  3. 03

    Implement the loop raw

    context → structured next step → execute → compact-append. No framework. Read the prompt you send.

    The Agent Loop
  4. 04

    Own the five

    Termination, sandbox, context builder, eval set, pause API. Review against the ready-to-build checklist.

    The Twelve Factors
  5. 05

    Add the thinnest library

    Only for a missing primitive: checkpoints, types, MCP, evals. If this is a coding agent, start from a harness, not a blank graph.

    Choosing a Framework

Done when

  • Golden set of 20–50 tasks exists.
  • The loop is drawn in four boxes and implemented in raw SDK calls.
  • Termination, sandbox, context builder, evals, and pause API are named.

Refuse

  • Eval by demo. 20–50 real tasks before the framework. promptfoo, Inspect, DSPy. CI on prompt change.
  • Unbounded loop. Force a done schema, a step cap, and a money cap. 'Done' is a tool or a structured intent.
  • Unowned prompt. Store prompts in git. Read what the framework sends. Eval on change.
  • Star-driven selection. Pick for the constraint: checkpoints, types, harness, documents. Read Canon's atlas.