Model comparisons

There is no best model, only the model that fits the job: the one that reads a panicked caller most faithfully may be slow and expensive, and the one that plans a good dispatch may be careless with free text. Every swap you make below trades something for something else - casualties against tokens, accuracy against latency, a better answer against a bill someone has to sign off on - so the work here is turning those trades into numbers you can point at instead of impressions you argue from.

Activity: A Different Dispatch Model

Dispatch currently runs on minimax-m3. Point DISPATCH_MODEL in your .env at one alternative and see how it fares. Replay the same scenarios on the original and alternative models enough times that you can tell a persistent difference from one lucky run. Record the model IDs, the total runtime tokens, and the evidence behind whichever you keep: a model that saves two casualties and triples the token bill is a trade you have to argue for, not a win.

Coordinate with your team to pick different models to evaluate.

What you will soon discover is that models, especially small ones, are acutely sensitive to the exact wording you give them: a line that steers minimax-m3 cleanly can leave a smaller model guessing. So experiment - tweak the prompt for the model you are testing, change one thing at a time, and re-run the same cases, and see what you can get.

When you’re done

As part of your evaluation, get your agent to report on:

1. the models you tested,
2. the total spend to clear all three maps on stage-5,
3. the model performance, and
4. the prompt tweaks you needed to get that to work.

Gather all the details from your teammates, and have one person’s code publish the writeup as an artifact. Post the link on the Padlet.

Activity: A Different Intake Model

Intake runs on mimo-v2.5, and INTAKE_MODEL swaps it the same way - the three alternatives above are worth trying here too, and the model that reads free text best is not automatically the one that plans best.

Unseal the instructor labels: uv run hadr-engine gen-datasets --include-labels adds contacts_labeled.jsonl (plus the reference report schema) to the same directory. This is a sizable dump of high-quality data you can use to measure the performance of your intake model. Having a lot of data helps average out the inherent randomness of using LLMs. Use the evaluation helper-prompt stub from yesterday.

Optional: intake on your laptop

If local IT permits, you can try running the models themselves on your local machine. Ollama and LM Studio both serve an OpenAI-style API from localhost that you can connect to your intake bot:

# Everything below the prefix is unchanged: dispatch still runs on OpenCode.
INTAKE_OPENAI_BASE_URL=http://localhost:11434/v1   # LM Studio: http://localhost:1234/v1
INTAKE_OPENAI_KEY=l337sp33k                        # any non-empty string; local servers ignore it
INTAKE_MODEL=qwen3.5:2b

Here are some candidates for size:

Here are some candidates for quantization. These are all the same Gemma 4 E4B weights, so any difference you measure is the quantization and nothing else:

Watch two things as you go down the list: whether the JSON parses at all, and which fields rot first. Quantization tends to corrode output when the model has to weigh multiple options or deal with uncertainty.

Coordinate with your team to pick different models to evaluate.

Ollama has a mechanism to force the output to be structurally valid JSON, but without a lot of tuning, your local models are going to be fast producers of confidently wrong fields. Tune your prompt alongside your model.

When you’re done

Get your agent to gather:

1. per-field accuracy against our reference data,
2. schema failure rate (how often the output is not valid JSON), and
3. wall-clock per contact. Local tokens are free, but you have to wait for them.

Gather all the details from your teammates, and have one person’s code publish the writeup as an artifact. Post the link on the Padlet.