Evaluation Infrastructure
Making LLM behaviour testable.
Pre-registered protocols · frozen reference sets · blind adjudication · pinned run records
LLM systems are easy to demo and hard to evaluate. A change can look better in a few examples while getting worse somewhere else. "It seems better" isn't enough to know.
I've been building evaluation infrastructure for an LLM decision system that reads records and makes calls on them.
Turning fuzzy behaviour into tests
The pattern is always the same. Take a claim nobody can check and give it a definition that can fail.
- The model understands the input
- Frozen labelled records, scored against a held-out set
- This prompt is better
- Two runs identical byte for byte except the prompt
- Personalization helps
- The same records, with and without user context
- This run is reproducible
- Model, prompt, context, spec and raw output pinned to the run record
What changed
The clearest result so far came from the extraction step, which is where the system reads a record and pulls out the facts it will reason about. I versioned the extraction specification and pinned a regression baseline to it, then measured how often a human adjudicator had to materially edit the output on records the system had not seen.
That rate went from 53% to 22% on the same frozen holdout. It measures this extraction process, not general model accuracy.
Keeping experiments honest
Protocols are registered before a run, so the question is fixed before the results arrive. Adjudication is blind when knowing the condition could affect the judgment. Holdouts stay at record level, and runs are immutable, so an old result can be reconstructed later.