aug 2026
a week ago i published the problem isnt the eval, its when it runs, and the honest summary of it was: i can see the trap, i cannot see the way out. the best evaluator we have is an llm judge, judges cost too much to point at everything, so everyone samples, and sampling is the worst instrument ever built for finding rare things.
i spent the week after that trying to design my way out of it. i got the answer from being told, fairly bluntly, that i had been working on the wrong problem.
two things happened close enough together that it took me a while to notice they were not the same complaint.
the first was a team who had thirteen quality metrics running on every agent call, and then switched all of them off, because a ten cent call cannot carry fifty cents of evaluation. i already believed that one. that was the memo.
the second was a room of editors telling me they reject llm as judge. not the price of it, the idea of it. and semantic similarity too, while they were at it.
i had a whole answer ready for this. it is conservatism, it passes, every new instrument gets this reception at first. then someone pulled up a page of their actual copy and i stopped talking.
the judge passes petrol station where the house style is fuel station. passes Woolworths where the house style is Woolies. it is not hedging on any of it. it is confident, and it is wrong in the one place a reader can check without any tooling at all.
so the objection was never really about judges. it was: your instrument is wrong about the things i can verify, so why would i trust it on the things i cant. and i had spent a week trying to make that instrument cheaper.
cheaper wrong is still wrong. you just get to be wrong more often per dollar.
the best line i heard all month is not mine and im going to keep using it anyway.
a system prompt tells the inmate not to kill anyone. a harness makes it impossible for them to buy a weapon. one of those is a rule and the other is a suggestion with good grammar.
i already believed this about agents. everything i ship sits inside a cage i built for it. tool permissions, schema validation, allowlists, hard limits, the whole boring layer, and ive rebuilt that layer more times than i want to admit to.
then the same idea got pointed at the thing doing the grading, and i did not enjoy where it landed.
an llm judge is an agent. running in production, at higher volume than the agents it grades, with a prompt and nothing around it. no version history, no monitoring, no path for a correction to travel down. it made a mistake on run one and it will make the same mistake on run four million, at full price each time.
i build a cage for every agent i ship and none for the one that decides whether the others worked. that gap had been sitting in my own architecture the whole time. it never occurred to me to look, because it never occurred to me that the evaluator was a thing that could be wrong in a way i was responsible for.
this is where everything i had been sketching rearranged itself, and it started as a vocabulary fix.
ive been saying python evals, semantic evals, llm evals, human evals. four kinds of eval. thats wrong, and it has been quietly costing me. the eval is the judgment. those four are instruments you use to reach one. one verdict per run, put together inside the evaluator out of whichever instruments that particular run actually needed.
which moves the interesting question from what are they to what is each one good and bad at, because that is the thing you can route on.
and then the asymmetry at the two ends does all the work. human is the only source of ground truth. python is the only one that enforces rather than opines. everything after this is a way of moving knowledge from the first into the second.
this was my answer a week ago, so ill be quick with it.
one evaluator, one conditional tree inside it. cheap instruments look at everything, the judge only wakes up for a run that earned it. every run goes in, every run comes out with a verdict and a record of the path it took.
i thought this was the interesting half. it isnt. it fixes what a verdict costs and does nothing at all about whether the verdict is any good, which is the thing the room full of editors was actually telling me.
heres the part i didnt have.
the deterministic layer is not only a cheap filter standing in front of an expensive one. that framing is exactly why i had been thinking of it as an optimisation. it does two different jobs and only one of them is about money.
the gate is the cost story. the override is the trust story, and it is the only honest answer ive got to what if your model is confidently wrong about my domain. the answer is that it does not get a vote on that part. terminology, house style, regional english, forbidden content, structure: those get decided by something that is incapable of having an opinion.
i own the jail.
now the part im least sure of and most interested in.
someone reads a verdict, disagrees, says why. today that goes into a slack thread and dies there. it should turn into something durable instead, and there is an order of preference for what kind of something.
the mechanism, stated flatly, because it improves over time otherwise sounds like it was written by a marketing department:
each correction turns a recurring fifty cent judgment into a one time free rule. the thing gets more accurate and cheaper in the same move, which those two almost never do.
the woolies correction is not a prompt tweak. it is a terminology rule. once it exists that entire class of error cannot happen again, and it costs nothing forever after.
it also answers a question i could not answer last week, which is who builds the routing tree. i had assumed a person sits down and authors it, and that assumption is precisely why the whole design felt so heavy. but a tree can be written out of a repo, from the prompts and tool contracts and rules that already exist. and it can be written by people disagreeing with verdicts. those are the two places the knowledge actually lives, and neither of them is a config page.
the sharpest thing anyone said to me about any of this was not about the engineering.
its not an engineering problem. the question isnt the engineering, its what does it cost to do that, whats the roi.
and thats right. loop optimising a judge is expensive. change the prompt, and to know whether you improved anything you have to re-judge a validation set. do that continuously and you have rebuilt the cost problem one floor up, in a nicer suit.
the answer is the ordering in that last figure, and it rests on one asymmetry i think is genuinely load bearing. deterministic corrections backfill. llm corrections dont.
a terminology rule can be replayed across a million historical runs for nothing. you find out immediately what it would have caught, and your history stays comparable to itself. a judge prompt change means re-judging everything you want to compare against, so it is expensive and it breaks comparability with everything before it.
then the bet, which is the part i could be wrong about. most corrections in a real domain are house style, terminology, regional english, formatting, forbidden content, structure. those compile to rules. if that holds, most of the loop is free and the roi question answers itself. if it doesnt, if most corrections need judgment to apply, then this is a much smaller idea than i think it is.
one rule, and it is the one keeping all of this honest.
a correction must never rewrite an evaluator on its own. a judge that edits itself destroys the base it was measuring against. the same scenario stops being evaluated the same way, no score is comparable to any other, and what you have left is an expensive random number generator with a dashboard.
so: correction, proposed version, a person approves it, new frozen version. scores never pooled across a version boundary. the backfill asymmetry is what makes those boundaries survivable, because a rule can be replayed backwards over the old version and a prompt cannot.
dispatch is dynamic. definitions are frozen. what changes per run is which instrument fires, never what it measures.
last week i listed three things i couldnt see through, and i want to be precise about which ones actually moved.
keeping a comparable baseline once routing is live. answered by the line above, and i believe that one.
recording skips so coverage gaps dont hide. answered by a change of framing rather than of machinery. a run does not skip an evaluator, it takes a path through one. if every run leaves with a verdict and the path it took, then a non run is a recorded state with a reason attached, not an absence rendering as green. still the hardest thing to get right in code, but no longer a hole in the design.
how much random audit you need. partly, and this is where i stop promising things. a small unrouted random slice exists to measure how often the cheap screens are wrong. it is not detection, it is calibration of the detector, and what it buys you is a number instead of a hope. so the claim is not we catch everything. it is know your number, and be able to prove it.
the last thing, and its why i went from stuck to fairly convinced.
the judge is rented. the harness is owned.
models change constantly, and right now a model swap throws away everything a team learned about evaluating their own domain, because all of it was living in a prompt. put it in rules and a model change doesnt touch it.
and the harness grows for free, out of people disagreeing with a judge, which they were going to do anyway and currently get nothing at all for. anyone can copy the routing in a quarter. nobody can copy a year of somebody elses disagreements.
which lands somewhere i wasnt expecting. evaluation is one of the capabilities i think you have to own rather than rent, and the reason isnt that models are bad at judging. its that the model is the rented part. the thing worth owning was never the judge.
shorter list than last week, but these are load bearing.
the person who knows the house style is an editor, not an engineer. if turning a disagreement into a rule means writing python, the loop never starts and none of the above happens. honestly i think this is the whole thing, and everything else is downstream of whether a non engineer can correct a judge in under a minute.
what actually compiles. the taxonomy of correction to artifact is what the entire design rests on, and its cheap to get wrong on a page and expensive to get wrong in code.
latency. it keeps coming up as one of the two things people actually care about and i dont evaluate it. neither does anyone else ive looked at. it doesnt fit the instrument frame cleanly, which usually means the frame is missing something.
a week ago my summary was that the best evaluator i have is priced out of the place it would be most useful. the sharper version is that it is also unsupervised, uncorrected, and exactly as good today as it was on the day i wrote it. routing fixes the price. it took someone showing me a page of my own domain to see that the price was the easy half.