Skill · pattern · evaluator-optimizer
Evaluator–optimizer
Generate, critique against a written rubric, rewrite, cap the loop. Use for writing, translation, code review, or any task with measurable quality where a single pass is not enough.
GET /api/canon/skills/evaluator-optimizer?format=md
Anthropic's eval-opt. Generator and evaluator must not share prompt or incentive. Cap iterations. promptfoo and DSPy make the rubric executable.
When
Clear scoring criteria and iterative gains. If you cannot write the rubric, you cannot loop it.
Do
- 01
Write the rubric first
Structured scores, not 'sounds good'. Include refusal cases.
- 02
Split generator and evaluator
Different prompts. Prefer a different model for the judge when stakes are high.
- 03
Cap the rewrite loop
Accept, revise (max N), or escalate to ask_human.
Don't
- Let the writer grade itself in the same prompt.
- Loop until 'perfect' with no cap.
Hard rules
- No rubric → no eval-opt. Use a chain or a human.
- The golden set is the spec. Architecture waits on it.
Refuse
- Eval by demo — Architecture first, golden set never. Success is a recorded GIF.
- Unbounded loop — No max-turns, token, wall-clock, or money budget. The agent rambles until the bill or the context dies.
Load with this
Load next
Trigger tests
Should fire
- “Add a critique-and-rewrite loop”
- “The draft quality is inconsistent”
Should not
- “The path is a known three-step extract”