aug 2026
this is the one im actually stuck on, so this is less a conclusion and more me writing out a problem to see it properly.
every observability platform ive looked at, including trodo which im building, gives you evals as a flat list. you get types. code evals in python or typescript, human evals, llm as judge, semantic ones. you pick which ones you want and the platform runs them. thats it. thats the whole model.
and the trouble shows up the moment you take it to production.
llm as judge is the best evaluator we have for most of what agents actually do. was this answer right, did it use the correct tool, did it hallucinate a field, did it miss the intent. a python assertion cant see any of that. a model can.
and its priced per call. at a few thousand traces a month thats a rounding error, so it feels solved. at millions, or billions, its an api bill that dwarfs the thing youre evaluating. you end up paying more to check the work than to do the work, which is not a tradeoff anyone signs off on, so they dont run it.
so the good eval exists, everyone knows its the good one, and almost nobody runs it on their real traffic.
the standard answer is: run it on one percent, at random.
think about what random sampling is good at. its good at estimating an average. it is structurally the worst possible tool for finding rare events, because rare events are rare, thats the definition. the failures that matter at scale are almost never the common ones. the common ones you found on day two. its the weird tail, the one prompt shape that breaks tool selection, the customer whose data has a field nobody anticipated.
one percent random sampling on a failure that happens once in ten thousand traces means you are, on average, not going to see it, and when you finally do you will have one example and no idea whether its a pattern.
so the industry answer to “the good eval is too expensive” is “run it where it is least likely to be useful.”
the other thing platforms offer is composition. take your python eval, your semantic eval, your llm eval, weight them, produce one composite number.
it took me a while to see why this doesnt help, and its about where the composition happens. a composite composes the outputs. which means every eval in it already ran. all of them, on every trace. you got a tidier number at the end and you paid the full bill to get it. thats the same cost problem wearing a nicer summary.
and even if compute were free it would still be wrong, because you cannot run the same eval for everything. a trace where the agent picked the wrong tool and a trace where retrieval returned nothing need completely different judgment. running every evaluator over every trace means most of them are scoring things they have no business scoring, and those scores go into the average anyway. you get noise, confidently weighted.
so composite is expensive and imprecise at the same time, which is an impressive combination.
heres the thing underneath the cost, and i think its the actual problem.
we accepted non determinism everywhere else in the stack. the agent picks its own path. it decides which tool to call based on what it just saw. we dont hand it forty tools and make it run all forty, that would be absurd, the entire point is that it chooses. the process is non deterministic by design and we built around that. we trace it, we accept it, we architect for it.
then we get to evaluation and it snaps back to a fixed list. same evals, same order, every trace, decided at config time before a single trace has arrived. the thing doing the work adapts. the thing measuring the work cannot.
why. if the agent is non deterministic and the process is non deterministic, why is the checking of it deterministic. thats the asymmetry, and once you see it you cant unsee it. cost is just the symptom that makes it urgent.
no eval can decide whether another eval runs. theyre siblings that dont speak. the result of one has no bearing on any other. thats the missing primitive.
what i want, and i dont have a clean way to build yet: run the cheap python eval on one hundred percent of traces, because its basically free. if it scores below some threshold, spend the llm call. if it lands in one band, run llm eval one. a different band, run llm eval two. let a semantic check decide whether the expensive judge is worth waking up at all. or let the model itself pick the evaluator, the same way it picks a tool.
same budget, aimed instead of scattered. cheap checks on everything, expensive judgment on the traces that earned it. thats strictly better failure detection per dollar than sampling, and its not close, because sampling picks at random and this picks on evidence.
theres a counter argument here and i want to hold it properly rather than wave at it, because it changes what you build.
your instrument should be more stable than the thing it measures. if the routing makes different choices this week than last week, you cannot tell whether quality dropped or the router just moved. you lose comparison, and comparison is most of what evaluation is for.
so the useful split isnt deterministic against non deterministic. its:
conditional. python scores 0.3, run llm eval two. same trace, same route, same answer, fully reproducible. the plan is dynamic, the result is not. your measurements stay comparable and you have given up nothing.
the model picks the evaluator. genuinely open ended, more powerful, and you trade away a stable baseline to get it.
i want both eventually. the first one delivers most of the value and costs you nothing you cant get back, so thats where i think this starts. the second is where it gets interesting, and you want the baseline problem solved before you go there.
if eval b only runs when eval a fires, then b’s pass rate is conditional on a. you cant say “the agent is 94 percent good” anymore, because that 94 is over a filtered population, and its not comparable to last month if the routing changed in between.
which drops me right back into a bug we already hit for real in trodo: an eval that didnt run has to be a recorded state, not an absence. an evaluator hit a limit, returned nothing, and looked exactly like an evaluator that ran and found nothing wrong. same output, opposite meanings, and the dashboard showed the reassuring one. one line of code, doing nothing, quietly.
a relational system multiplies that by every branch you add. every non run is now a decision somebody made, and if it isnt recorded as “skipped, because a scored 0.8” then your coverage is silently full of holes that all render green. you need the reason, not just the result. i think thats the actual hard part of building this, harder than the routing.
the other hole. how do you know the cheap trigger has decent recall?
if the python check never fires on some class of failure, the llm eval never runs on it, and you never find out that class exists. the router is only as good as the dumbest thing in it, and the failures it misses are invisible by construction, which is the worst kind of missing.
so sampling doesnt die. it changes job. you still run some random llm evaluation, not as primary detection but to audit the router, to answer “what is my cheap layer not catching.” thats a completely different question from “how is the agent doing” and it needs far less volume to answer.
and the obvious regress: if a model chooses the evaluators, evaluation is an agent now, and you have to evaluate the evaluator. i dont have a clever answer. the only stopping point i can see is that the cheap deterministic layer is the ground you stand on. it doesnt get judged by another model, it gets judged by being simple enough that a person reads it and says yes, thats right.
evaluation is one of the capabilities i think you have to own rather than rent, and this is the part of owning it that i havent solved.
im fairly convinced the missing primitive is that evals cant talk to each other, and that the fix is composing execution rather than composing scores. thats the direction were pushing trodo in, evals that run at any scale and any level because the plan adapts to the trace instead of being fixed before it arrives. saying it out loud is the easy half. what i dont have is:
how you keep a comparable baseline once routing is live. how you record skips well enough that coverage gaps show up instead of hiding. how much random audit you need to trust a cascade, and whether that number is knowable in advance or only after something slips through.
and honestly whether people want this. it is more moving parts, and more moving parts is how you get a system nobody can reason about at 3am. flat lists are dumb but you can hold one in your head.
but the current situation is that the best evaluator we have is priced out of the place it would be most useful, and the workaround is to run it where its least likely to help. that cant be the end state. the problem was never which evals exist. its that all of them run the same way, always, on everything.