Vol. X · Field Operations · X.7
Cost, Cache, and Model Routing
Tokens are inventory. Prefix cache, model routing, and 'do not resend the PDF' are design, not FinOps after the bill. A correct small model on a tight window beats a frontier model on a dump.
6 min read
Doctrine
- Stable prefixes (system, skill index, constitution) should be cacheable. Do not shuffle them every turn.
- Route easy hops to small models. The classifier is itself a cheap call.
- Tool results go to files; the window gets a pointer. That is a cost control and a quality control.
The cheap architecture is the good one
Routing is Anthropic's catalog and also a bill. Prompt caching rewards a stable layout (Factor 3). Compaction (Factor 9) is how hour-three still fits. DSPy and promptfoo exist so you can measure whether the cheap path still scores. Vercel AI SDK and LangChain both make routing ordinary code. Use them for that, not for hiding the loop.
Anti-patterns
- Rebuilding a 20k-token system prompt with a timestamp so the cache never hits.
- Frontier model for every classification.