Presentation and finale

Day 2

Presentation and finale

Publish what you built, show one thing you learned building it, then prove it on the challenge map.

Your solution page

Everyone publishes their own page on GitHub Pages, from the agent’s own repository: the repo that does the work also explains it.

Required content:

  • The intake skill: how natural-language contacts become typed reports, and the evidence that your chosen model preserves accuracy.
  • The reconciliation process: your matching rules and thresholds, and how conflicting claims survive.
  • The dispatcher loop: standing orders, prioritization, and how new evidence changes an earlier decision.
  • The stores: reports as append-only evidence, incidents as mutable beliefs.
  • The evidence: model comparison results and your baseline against the greedy dispatcher.

The quiz gate

Before you publish, prove you understand your own system. Ask Claude, with this prompt exactly:

Give me an HTML report on this system with context and intuition, and a five-question quiz at the bottom. I will not present until I score five out of five.

Take the quiz on your own page. Fail? Read properly, retake. It happens to all of us.

Activity

  1. Have Claude generate the page from your repo - it knows what you built, so make it explain why.
  2. Tell Claude to wire up GitHub Pages internally, so the page publishes from the same repository as the agent.
  3. Review every word and make it yours. The why is the part an agent cannot write for you.
  4. Post the link on the Padlet - the GitHub Pages page, not an artifact.

The test: someone who has never seen your repo should understand your agent without you in the room.

Show and tell

Time decides the format; the instructor calls it on the day.

  • If there is time: a team walks the class through its loop from its published page.
  • If there is not: each person takes ninety seconds for two things - the one tweak, change, or design decision that made the biggest difference to your score, and one thing Claude did that surprised you.

Either way, bring the number that proves it: what the score was before the change and after.

The finale

Each group runs its agent on the finale scenario through the complete outer runner, without operator intervention. The finale is one of the stage-5 scenarios (stage5-triage@londone, stage5-triage@stockhome, or stage5-triage@feeladelphia), announced by the instructor on the day; the maps and scenarios are known in advance.

Stage 5 is beatable outright: a careful dispatcher can finish it with zero casualties and zero buildings lost. If you are losing a building, the problem is your triage, not the scenario.

Before the run

  • Freeze the group branch, configuration, runtime model IDs, and assigned seed.
  • Run deterministic checks from a clean worktree.
  • Confirm that no extraction labels, hidden truth, or end-of-episode metrics reach the agent’s context during play. The engine’s own display shows the live score on the wall, but your agent never reads that surface - if your code does, that is the bug this check exists to find.
  • Confirm the episode stops when no fires remain and none are scheduled, or at its tick limit.

Use the finale helper-prompt stub as the placeholder for the last readiness check.

Submission

  • Casualties.
  • Buildings lost.
  • Self-reported total runtime tokens.
  • Intake and dispatch model IDs.
  • A short demonstration of one observation, reconciliation, and replan from the run.

Discussion

Three rounds, with the finale numbers in front of you. Argue each one out before opening its spoiler - those are where the discussion tends to land, not the answers.

Score against budget

  • What is a casualty worth in tokens? Put a number on it, then defend it.
  • Where did your tokens actually go - intake, dispatch, or retries? Did you guess right before you looked?
  • Name a change that improved the score and cut tokens. What made it different from the ones that only did one?
  • Whose scores would you trade for whose token totals?
  • Which constraint would bind harder in a real deployment, and does that change what you would have built?
Where this usually lands

The room will not agree on what a casualty costs, and that is the finding: without a stated exchange rate, “optimize the agent” has no meaning, and every group has silently picked one. The cheapest wins are almost always structural rather than model-side - work moved out of the context window, a tool call that did not have to happen, a state message that stopped shipping the whole world every tick. A change that improves the score and cuts tokens usually did so by giving the model less to be wrong about.

Picking models with evals

  • Which model won for each role, and on which metric? Would a different metric have flipped it?
  • How many samples did you look at before you believed the result?
  • What did the cheaper model get wrong - the same thing every time, or something new each time? Which of those is easier to engineer around?
  • Could you have picked the winner without running the eval? Would you have been right?
  • What did the finale expose that your eval never measured?
Where this usually lands

Somebody’s ranking will flip when the metric changes, which is the point: the eval encodes a choice about what matters, so disagreeing with a result usually means disagreeing with its metric. Sample counts tend to be embarrassingly small once said out loud. The most useful distinction is systematic error versus random error - a model that is wrong the same way every time can be prompted or post-processed around, one that is wrong differently every time cannot. Expect the finale to surface something no fixture covered; that gap is the argument for running both offline evals and live episodes rather than trusting either alone.

Separation of concerns

  • Which parts of your system decide, and which only report? Where is that line drawn, and what happens if something crosses it?
  • Which parts are deterministic and which are model calls? Would you move any of them now?
  • contact_id and reported_location come from the envelope, never from the model. Where else does that argument apply in what you built?
  • If you had to swap the intake model tomorrow, what else would have to change?
  • What did you end up fixing in two places? What should that have told you?
Where this usually lands

The shape the course is pushing: models propose, deterministic code disposes. Intake claims, reconciliation decides, the stores hold evidence separately from belief, and the envelope - not the model - owns identity and location, because those are facts about the contact rather than judgments about its content. The test of the boundary is the swap: if replacing a model forces changes outside its own module and prompt, the boundary was in the wrong place. Anything fixed in two places was one concern living in two.

Close

Discuss where the same inner loop, outer runner, deterministic policy, and eval separation apply at work. On-prem models and scheduling are reference topics, not live exercises.

From idea to action

To close, here’s (part of) a talk by Anthropic’s Thariq Shihipar that serves as a final reflection of how the work changes.