Evals are easy to postpone.
Expensive to skip.

From demo to discipline

“It feels smarter” hides the tradeoffs.

Model changes rarely improve everything at once. A candidate can follow instructions more closely and cost five times as much. A smaller model can be faster on ordinary cases and brittle when the evidence conflicts. A new prompt can produce more persuasive prose while quietly becoming less grounded.

Those tradeoffs get harder to see as an AI feature becomes a workflow. Consider a product that finds opportunities, ranks them, prepares an artifact, reads a communication, and proposes a next action. Every step depends on a different kind of truth:

• Discovery cares about source freshness and canonical identity.
• Ranking cares about grounded evidence, explicit dealbreakers, and bounded scores.
• Artifact generation cares about relevance without invented facts.
• Communication tracking cares about entity matching, stage, urgency, and the right next action.

One global benchmark cannot tell you whether that product is getting better. The unit of evaluation has to follow the unit of work.

Evaluation architecture

Let each workflow lane define “good.”

The goal is not to crown one model. It is to decide which route is reliable enough for each task.

Start by writing the failure you actually care about. A discovery lane might fail by preferring a stale aggregator over the official source. A ranking lane might return valid JSON with an invalid record ID. A document lane might add an attractive but unsupported claim. A communications lane might connect an interview email to the wrong opportunity.

Each failure suggests its own fixture, evaluator, and review rule. The rubric can then combine several kinds of evidence:

Correctness

Did the output make the right decision for this case?

Grounding

Can the conclusion be traced to the provided sources?

Schema reliability

Did the system return the structure downstream software needs?

Failure behavior

Did it stop, hold, or ask for review when the evidence was insufficient?

Operating fit

Are latency and cost appropriate for this lane and volume?

Operating model

Spend in layers: static, dry run, then live.

An evaluation harness should make the cheapest useful failure happen first. That keeps routine development fast and reserves model spend for questions that only a live provider can answer.

1. 01

   Static

   Validate configuration, environment wiring, fixture completeness, schemas, evaluator contracts, and call-site readiness without calling a provider.

2. 02

   Dry run

   Build the exact payloads, exercise the harness path, estimate cost, and preview the comparison artifacts. Still no live model spend.

3. 03

   Live

   Run only with an explicit candidate, budget cap, and saved fingerprint for the model, prompt, tools, fixtures, and evaluators being measured.

This separation catches bad wiring and broken evidence early. It also keeps “evaluation” from becoming a euphemism for an unbounded batch of API calls.

Review and forensics

A score should be the start of the review.

Pass/fail alone is too thin to debug. When a candidate regresses, a reviewer needs to see the fixture, the expected behavior, the actual output, the evaluator’s reasoning boundary, and the source evidence behind the decision.

A practical run can leave several views of the same evidence:

• Run dashboardLane outcomes, candidate comparisons, spend, latency, schema failures, and the current recommendation.
• Fixture workspaceExpected behavior, reviewer annotations, ambiguous cases, and the next fixture to add.
• Record drilldownInputs, outputs, evidence excerpts, individual scores, and a human-review signal.
• Decision logPromote, continue evaluating, or hold—with the evidence and fingerprint that produced the call.
• Portable artifactsJSON for replay and forensics, CSV for analysis, and integration config for the team’s chosen runner.

Keep operational metrics boring.

Observability works best when its labels stay bounded: lane, provider family, candidate label, evaluator, status. Fixture IDs, input text, URLs, thread IDs, artifact paths, and failure prose stay out of metric labels. Put that detail in the saved run evidence, where it is useful without creating a cardinality problem.

Release decision

Promotion is a product judgment with rules.

The most useful output is not a leaderboard. It is a defensible next step.

A candidate can be promoted when it clears the lane’s safety and schema gates, improves the behavior that matters, and makes sense on cost and latency. A near tie should usually become “keep evaluating,” not a forced winner. A cheap candidate can remain the default for ordinary work while a stronger model is reserved for ambiguity. A safety or grounding regression should hold the route even if the average score is higher.

That conservatism is a feature. The harness should make uncertainty visible rather than manufacture certainty from a small test set.

A practical first version

Start with the decision you are afraid to get wrong.

You do not need a general evaluation platform before you can measure useful behavior. Pick one consequential workflow and build the smallest fair harness around it:

1. Name the decision.What will the system choose, create, classify, or do?
2. Write five to twenty cases.Include ordinary inputs, ambiguity, missing context, conflicts, and a case that should stop for review.
3. Separate hard checks from judgment.Use deterministic validation for identity, schema, required content, and prohibited actions. Use rubric or human review for nuanced quality.
4. Save the evidence.Keep the input, output, scores, cost, latency, fingerprint, and reviewer note together.
5. Repeat before declaring a winner.Measure variance and treat small margins as uncertainty.

That first harness will not answer every model question. It will answer a real product question repeatedly—and expose where the next fixtures belong.