jun 2026
ive rebuilt the way i build agents about five times now. not tweaked, rebuilt. each time i was fairly sure the new way was the right way, and each time something broke it, or broke the assumption sitting underneath it. the current one is working. is it the best, i genuinely dont know. so far so good.
im writing the phases down because i think the useful part isnt the architectures. its what killed each one.
the first instinct is obvious. the model cant do things, tools can do things, so wrap every capability you have as a tool and hand the model a big list. every use case becomes accessible. it felt like progress because it was legible, you could point at a list and say look at everything my agent can do.
what killed it was the list. tool selection gets worse as the list grows, not linearly either, it falls off. and every tool description is context you are spending before the task even starts. i was paying a large fixed cost on every single call to describe forty things so the model could pick one. at some point i was debugging tool choice more than i was debugging the actual work.
so route it. use something small and cheap to decide which tool, and save the expensive model for the consolidation and the analysis at the end where it actually matters. on paper this is clearly correct. most of the calls in an agent loop are boring.
it broke in a way i didnt expect. routing errors turned out to be the worst class of error i had. theyre cheap to make and expensive to notice. a bad piece of analysis looks bad, you catch it. a bad tool choice just quietly sends the whole run down a wrong path and everything downstream is confidently working on the wrong thing. and when it goes wrong you retry, which means you end up paying for the big model anyway, plus the small one, plus the wasted work in between.
i also just got repriced. the gap between cheap and good narrowed while i was busy optimising for it.
then the swing to the other extreme. stop giving it tools, give it a sandbox. let it write code, let it write its own queries, recursive language models composing whatever they need. one tool instead of forty. it felt like i had found the actual answer, because it dissolved the phase one problem entirely instead of managing it.
what i learned is that unbounded capability is an unbounded failure surface. it can do anything, which means it can do anything wrong, and the space of wrong is much larger than the space of right. constraining it after the fact is much harder than never handing it the capability. its brilliant when it works and its very hard to reason about when it doesnt.
this one didnt die, and i want to be explicit about that, because the pattern of this piece is things breaking and this is the one that didnt. giving an agent continuity changes what it is. memory is still the centre of what i build. it raised a question about what shape it should have, which i get to further down, but the idea held.
what ended the phase was not memory failing. it was noticing what my memory was sitting inside.
everything i had built quietly assumed i was on a big closed frontier model and would stay there. that assumption was invisible to me because it had never cost me anything. and then the open models got good. kimi and the ones after it went from interesting to genuinely competitive with the closed frontier on the actual work i do. not on everything, and not always, but close enough often enough that the question stopped being which frontier model and became why am i structurally unable to use anything else.
that was the uncomfortable answer. i couldnt move. not because the alternatives were worse, but because my memory lived inside one vendor and everything above it had grown around that vendor’s shape.
and thats when the framing changed for me. being able to change model is not a preference, its a moat. if you can move and your competitor cannot, you collect every price drop and every capability jump the week it ships. they collect it whenever their provider decides they should, at whatever price the provider sets, and they pay a migration to get anything else. that gap compounds, quietly, in your favour, and it costs almost nothing to hold open if you build for it from the start.
the current one is the first phase that isnt really about the model at all. its about the architecture, and about how many capabilities i can actually build myself.
because thats the thing i had backwards for four phases. i kept asking where the model should go, what it should be allowed to do, how to work around what it couldnt. the better question is: what capabilities does this outcome need, and which of them am i going to own.
memory is one. it goes to supermemory or mem0 or whatever else is good, as a thing i run, not a thing the provider runs for me. a filesystem is another, laid out so it can find the right tool instead of being handed all of them. skills for approaches, so its not rederiving how to do something every time. harnesses at every level, so when something breaks it gets caught at that level and not at the very end, which is the difference between a bug and a mystery. retrieval, evaluation, the record of what happened. each one is its own component with its own boundary.
the model is then one component among those. an important one, it does most of the actual thinking. but a component, and the one i want to be able to rip out.
which gives me a test i can actually run. swap the model for another one of roughly equal capability. does the whole thing still work. if the answer is no, some capability i thought i owned is actually living inside a vendor, and i should go find it.
so the model agnosticism isnt really the goal. its just what happens if you build the capabilities properly. you get it for free by owning your own pieces, and you lose it the moment you let one of them quietly move into someone elses product.
almost none of this is model work. its layout, boundaries, and where you put the checks.
and it costs me something, which i should say plainly. provider native features are usually better than the portable version. theyre tuned to that model, theyre cheaper because theyre bundled, theyre one less thing to run and pay for. every time i pick the third party version im paying an abstraction tax and betting portability is worth more than integration. that bet can absolutely be wrong. if one provider pulls far enough ahead that its native memory is just categorically better, ill have built a very careful bridge to nowhere.
the other honest bit is that “equally capable” is doing enormous work in that sentence. models are not drop in replacements. they differ on tool calling, on how literally they follow instructions, on what happens when the context gets long. so model agnostic is a direction i push in, not a state i have arrived at. every swap ive done has broken something small and stupid. its less breakage than being locked in. its not zero.
lining all five up, the pattern is the same one every time and i missed it for years.
most of what i built in phases one through four was a bet on the models weaknesses at the time. toolifying was a bet that it couldnt write code. routing was a bet that intelligence was expensive. the sandbox was a bet that it could handle freedom better than it actually could. none of those bets were stupid, they were correct on the day i made them. they just all expired.
so it isnt that i kept getting the architecture wrong. its that a lot of what you build around a model is scar tissue around whatever it couldnt do that year, and scar tissue depreciates. the frontier moves and your clever workaround becomes dead weight you now maintain and eventually rip out.
which means the question i should have been asking the whole time isnt “what is the best architecture.” its:
which parts of what im building are betting on a weakness that gets fixed in six months, and which parts are actually mine.
the harness is mine. the definition of what a good outcome even is, thats mine. the evaluation, the domain knowledge, knowing which failures actually matter to the person using this. those survive a model upgrade. tool routing logic does not. prompt scaffolding does not. half my framework choices did not.
and thats really what phase five is doing. its not a smarter architecture than the other four, i want to be careful about claiming that. its that ive stopped designing around what the model cant do and started designing what i want to exist, capability by capability, and then letting the model be one of them.
partly because i cannot predict which capability lands next, so the only thing left to optimise is how cheap it is for me to change my mind. not “this is right”, but “when this turns out to be wrong, what does it cost me to move.” partly because being able to move is worth more than being on the best thing at any given moment, and that only stays true if you keep it cheap to move. and partly because the capabilities i build are the only part of this that compounds. every phase before this one, i threw the work away when the frontier moved. memory i own, evaluation i own, a harness i own, those carry forward into whatever phase six turns out to be.
which is also what i mean by agents being more than models. not that the model is a small part, it does most of the actual work. its that the model is the part everyone has, the part that changes under you without asking, and the part you should be least attached to. whats yours is everything around it that survives the swap.
the specific thing i got wrong in phase four, and i think this generalises.
structure is a liability for facts with a short half life. a schema encodes what mattered on the day you wrote it. if the world moves faster than your schema, you spend your time migrating the schema instead of using the information, and worse, the schema quietly filters out anything it didnt anticipate. unstructured plus good retrieval wins for that category, which is one of two reasons the third party memory tools have a real job.
the other reason is the whole point of phase five. memory is the single most expensive thing to move once its full. if it lives inside a provider, then the day you want to switch models you are not switching a model, you are migrating your agents entire history, and you will not do it. you will stay. thats not a model choice anymore, thats a lock in you built yourself and called convenience.
but it costs you something and i want to be honest about it. unstructured memory is not auditable. you cant diff it. you cant look at it and tell what it silently dropped. structured memory is rigid and wrong in ways you can see. unstructured memory is flexible and wrong in ways you cant. thats a real trade and i dont think its resolved, im just currently choosing the second one for anything that changes weekly, and keeping structure for anything i need to be able to argue with later.
i kept saying designing agents is architectural, more art than technical, and honestly when i first said it i was hand waving. so let me try to make it mean something.
architecture is the set of decisions that are expensive to reverse.
thats it, and it sorts everything immediately.
its the same split i wrote about at the scale of a company, where the expensive decision is the shape of the process and the cheap one is the tool you bolt to it.
almost every agent argument i see online is about the cheap list. which framework, which model, which orchestration library. those are real questions and theyre also the ones you can change on a wednesday afternoon. nobody argues about the record shape and thats the one youre stuck with.
except, and this is the bit that took me five phases, model choice is only a cheap decision if you did the architectural work to make it one. if your memory lives in the provider and your tools are shaped around one models quirks, then the model isnt a cheap decision at all. its quietly the most expensive one in your stack and you made it in an afternoon off a benchmark. thats the same decision sitting on either list depending entirely on choices you made somewhere else.
the neat ending here is: define the outcome first, then work backwards, pick the destination before the vehicle. and i do believe that. its how i work now, and phase five is the first time it has an actual mechanism behind it rather than being a nice sentence. outcome, then which capabilities that outcome needs, then which of those i own. the model shows up somewhere near the end of that list instead of at the start of it.
but i have to be honest that the essay above is five rounds of me picking the vehicle first. i did not arrive at outcome first by thinking clearly about it. i arrived at it by doing the opposite five times and watching each one expire. so when i say its common sense, we choose a destination and then think about the road, thats true, and its also not what i did, and im not sure the lesson transfers without the five rounds. maybe you have to burn a few architectures to stop being seduced by them. maybe reading this saves someone one round. probably not five.
and the frontier is still moving, faster than any of this. which means phase five is not the answer either. its just the one that hasnt expired yet.
so far so good.