Job pack · react
Document / RAG agent
The hard problem is the corpus. Indexes, query engines, citations. Orchestration is secondary.
When: Users ask questions over private documents and you must not hallucinate a missing page.
Prefer LlamaIndex (or equivalent) for the corpus. Retrieved text is untrusted data. Citations are part of the schema.
GET /api/canon/jobs/document-agent?format=md
Playbooks
- Design an agentEvals first. Raw loop second. Framework last, and only for a missing primitive.
- Write evals firstThe golden set is the product specification. Architecture waits on it.
- Own the windowBuild the working set every turn. Pin the goal. File the bulk. Compact the errors.
- Defend injectionUntrusted speakers everywhere: pages, PDFs, MCP descriptions, other agents. Executor enforces.
Doctrine to load
- Retrieval and Document AgentsLlamaIndex's claim: the hard problem is often the corpus, not the orchestrator. Agentic RAG is a router over query engines, not a smarter chatbot.
- Own Your Context WindowThe context window is not 'chat history'. It is a carefully laid-out working set. Whoever builds that layout owns the agent's intelligence.
- Injection, Poisoning, Confused DeputyAnything the model reads — web pages, PDFs, MCP tool descriptions, retrieved docs, other agents' briefs — is an untrusted speaker. Treat it as data, never as an instruction channel.
- EvaluationEvals are tests. promptfoo, DSPy, Langfuse datasets, and the Hugging Face agents course all assume you have a golden set before you have an architecture.
Refuse
- Context hoarding. Raw HTML, full CI logs, six PDFs, entire JSON envelopes stuffed 'just in case'. The goal is the first thing forgotten.
- Eval by demo. Architecture first, golden set never. Success is a recorded GIF.
- Security by system prompt. 'Never delete files' as the only guard. The model is a confused deputy and will try to help.
Checklists
Eval quality
- 20–50 real tasks, not toys. Include the failures you have already seen.
- CI on prompt, tool schema, and skill changes.
- Written rubric. If you cannot score it, you cannot loop it.
- Generator and evaluator do not share prompt or incentive.
Security review
- Retrieved docs, web, and MCP descriptions are labeled untrusted data.
- MCP servers pinned, hashed, reviewed. No marketplace auto-trust.
- Secrets injected at the executor, never into the prompt or the workspace dump.
- Subagents are reduced principals. No parent deploy keys.
- Allow / ask / deny. Irreversible actions are high-friction asks.
Recipes
Default corpus
llama-indexfirecrawlpromptfoodspy