Subagents and Evaluation

Day 1

Subagents and evaluation

Ask the same model the same question twice and you can get two different answers, both plausible. So, how do you test a model?

We’ll explore that question with subagents here, by building evals for our intake model. Evals are a set of test cases engineered to evaluate the behaviour of an LLM. In this case, the evals consist of input and output pairs, where we can use an LLM to judge if the input and output are correctly paired.

Our intake model is responsible for reading each incoming contact - a panicked call, an alarm, a responder update - and turning it into the structured report the rest of your dispatch system runs on. Get it wrong and everything downstream is wrong with it.

Get the raw, unlabeled data with uv run hadr-engine gen-datasets: it writes datasets/contacts_unlabeled.jsonl. This will be our input set – the rest of this section is how we obtain the output set from this.

Look at that file before continuing.

Goal: Extract what the user said

Your dispatch code is code, not an LLM. It needs structure in the data to work. That’s where the intake model comes in: what kind of incident, how bad, how many people, what evidence the caller actually gave. Its job is to extract what was said, not what is true.

A caller can name the wrong building or exaggerate, and that is the simulation’s problem, not intake’s. The envelope already gives you the machine-readable reported_location and occurred_at, so those are copied, never re-derived from the words.

For example, this line of the file - note that every row wraps its contact in a contact key, so read row["contact"], not row:

{
  "contact": {
    "contact_id": "unlabeled-con-1",
    "delivered_at": 6,
    "occurred_at": 5,
    "payload": {"text": "Listen, by the harbour market <l139>, 2 people definitely still in there and really bad, there's a fire. There's smoke."},
    "payload_type": "human_text",
    "reported_location": {"building_id": "l139", "kind": "building_id"},
    "source_type": "public_call"
  }
}

might become:

{
  "incident_type": "fire",
  "severity": "high",
  "headcount": 2,
  "evidence": ["smoke", "flames reported"]
}

What fields, what values they can take, how to mark “the caller said nothing about this” - that schema is yours to design, and the activity below will pressure-test it against 300 contacts.

Subagents

A subagent is a separate Claude with its own context window and its own tool limits; it does a job and returns only its result to the main session. That separation offers four benefits:

  1. With a fresh context, it is not hindered or confused by the past of the main session
  2. It keeps the main context lean, only returning the answer (and not the whole exploration process).
  3. It allows you to assign different tasks to different models, saving money on things that can be done cheaper.
  4. It allows you to parallelize processing, making things quicker.

Careful: an unbounded fan-out can spawn dozens of subagents in parallel and burn through your plan limits fast, so cap it.

Activity: Discovering Structure in Data

From the data, we have 300 unlabelled data points. We need to simultaneously

  1. find the structure in the data,
  2. extract the data using that structure, and
  3. assure the quality of the data.

This corresponds to:

  1. Finding a common schema,
  2. Building an extraction SKILL, and
  3. Building a rubric to judge extractions.

(In this context, a rubric is a skill that is used to judge LLM output.)

Even for a human, this is a daunting task, requiring a lot of patient trial-and-error. With the right structure with AI, we can do this rapidly.

The pattern: subagents + adversarial review

The basic strategy is to fan cheap extractors over the data, then have an independent critic judge all of it. The critic half echoes the adversarial PR review from this morning’s plan, build, review - a reviewer that did none of the work judges all of it. Reach for this sort of pattern whenever there is more data than you can read and an extraction process you cannot yet trust.

Warning: this loop burns tokens fast - every cycle is a fleet of annotators plus a critic pass - so only one student per group runs it. This is the one activity in the course you do not each do in your own repository. Pick a driver, have them share their screen, and everyone else watches the loop run and reads the intermediate files as they land: the schema argument, the rubric, and the labels the critic rejects are the lesson, and you only see them live. Do not start a second fleet in parallel to follow along.

Note: If you get stuck here, try running the helper prompt and asking it to refine your prompt.

flowchart LR
    U[("contacts_unlabeled.jsonl")] --> H1 & H2 & H3
    H1["Haiku annotator 1"]
    H2["Haiku annotator 2"]
    H3["Haiku annotator n"]
    H1 & H2 & H3 -- "labels + skill" --> C["Sonnet critic - never an annotator"]
    C -- "rubric: accept/reject each label" --> G{"95% accepted, or 3 cycles?"}
    G -- "no: re-run with the synthesized common skill" --> U
    G -- "yes" --> F[("accepted labels = Day 2 fixtures")]
    C -. "proposed schema extensions" .-> S["your report schema"]
    H3 ~~~ S

Step by step:

  1. Fan out annotators. Have Claude spawn several Haiku subagents, each with its own slice of contacts_unlabeled.jsonl and your report schema. Each produces:

    1. a full extraction of its slice of the data independently, and
    2. also writes its own skill: the procedure it found itself following, mistakes included.

  2. Adversarial review. Spawn one Sonnet critic that did no annotating. It reads every label and every skill and returns three things:

    1. a rubric of explicit accept/reject criteria, applied to every label;
    2. one common skill synthesized from the annotators’ drafts, keeping what they agreed on and settling what they did not; and
    3. schema extensions - the critic’s brief is to maximize the useful information extracted, so it gets your schema and proposes fields for anything callers keep saying that the schema has nowhere to put.

  3. Loop until accepted. Re-run fresh annotators with the synthesized skill and let the critic re-judge.

The driver tells their Opus agent to run this loop with /goal, with a success criterion (~95% of extractions accepted) and an escape hatch (no more than 3 cycles). Goals do not stop well on their own, so every goal gets a cap. Also tell it to write all intermediate states to disk, so the group can inspect them afterwards and so the driver has something to hand over at the end.

Some important observations:

  1. the cheap model sits closest to the data and touches every contact, while the pricier model never reads the raw data at scale - it reads the distilled outputs and spends its tokens on judgement.
  2. you can judge the quality of extraction by looking at the skill and rubric far quicker than by looking at hundreds of examples.
  3. This sort of extraction is only possible because of the adversarial review structure.

The key lesson of this section: clever structure gets you far with AI.

When you’re done

One person ran this, but everyone needs the output: we’ll be using them on day 2, in Stage 2: intake. So the driver:

  1. Runs npx ccusage, screenshots it, and posts that on the Padlet. That is what the whole group’s loop cost - compare it against the other groups.
  2. Opens a pull request against every teammate’s repository carrying the same four things: a. the prompt that drove the loop, b. the synthesized skill, c. the rubric, and d. the accepted labels.

Each repository owner reviews that pull request and merges it themselves - same rule as every other pull request, and it is worth reading rather than rubber-stamping, because you are about to build on it. If the critic proposed schema extensions, that review is where you decide whether to take them.