I opened the billing page and saw zero. Not "running low" - exactly zero. The culprit wasn't some complicated scenario or a flood of users: one scheduled agent spent about 14 hours working with itself and made 129 language model calls. The task it was solving was deterministic - it could have been closed with a script and an ordinary condition.
This is an article about why I stopped treating the unreliability of an agent system by adding more agents. Below: what exactly broke, which layer turned out to be dead weight, and the rules that replaced quantity - a call budget, stop criteria, a retro after every task, and memory under version control. And why it's more reliable to check a result with one skeptic than with a crowd of judges.
A couple of terms up front. An agent is a program that decides in a loop which step to take next, and asks a language model about it. A call is one request to the model, which you pay for. The rest I'll explain as I go.
The lure of the swarm: why you want to add one more agent
The logic of adding an agent always looks convincing. The result came out crooked - let's add an analyst who'll break the task down before work starts. The analyst got it wrong - let's add a skeptic to check them. The skeptic argues with the executor - let's add an orchestrator to settle it. Each step feels like cheap insurance: one more pair of eyes, one more layer of protection.
The problem is that this pair of eyes is nondeterministic. A new agent isn't a check, it's another source of decisions that can be wrong, loop forever, or wander off. Layers of protection add up to reliability only when each layer is more reliable than what it's checking. If you put a probabilistic controller over a probabilistic executor, you don't get a guarantee - you get a second probability and twice the bill.
The second trap is the sensation of work. A swarm produces a lot of visible activity: discussions, plans, intermediate documents. From the outside it looks like the system is thinking. In practice it can burn the entire budget talking about a task it hasn't started yet.
The incident that put everything in place
An agent with a language model sat on a schedule and woke up every 30 minutes. Over roughly 14 hours it made 129 calls and zeroed out the balance.
The arithmetic matters here. There were about thirty wake-ups in that time, and 129 calls. So one wake-up is not one call: inside each run, the agent also reasoned about what to do. When you put an agent on a schedule, you're multiplying not the number of runs, but the number of runs by the average length of the internal loop. You usually don't control that second number and don't hold it in your head - and it's exactly what turns "once every half hour, that's pennies" into an empty account by morning.
I wrote the conclusion down for myself in capital letters: a deterministic check is a script, not an agent. If the condition can be expressed as "if A, then B," you don't need a language model there. It doesn't add intelligence, it adds cost and variance. You need a model where the input is unstructured and it isn't known in advance which steps will be required. A scheduled state check is not that case.
And a separate lesson about limits. A scheduled agent has no natural end: a human closes the laptop, a loop doesn't. Anything that runs without a human in the loop needs a boundary set from the outside, not a hope that it will stop on its own.
flowchart LR
subgraph BEFORE["Было - агент на расписании"]
A1["Пробуждение каждые 30 минут"] --> A2["Внутренний цикл рассуждений"]
A2 --> A3["129 вызовов за ~14 часов"]
A3 --> A4["Баланс обнулен"]
end
subgraph AFTER["Стало - скрипт плюс бюджет"]
B1["Проверка условия скриптом"] --> B2["0 вызовов модели"]
B2 --> B3["Модель только там, где вход неструктурирован"]
B3 --> B4["Не более 5 вызовов на задачу"]
endThe dead layer: 63 files nobody ever opened
I found the second symptom of the same disease not in the invoices but in the repository. The system contained a domain layer of 63 questionnaire files - structured descriptions of the subject area that agents were supposed to fill in and then reuse. They were built "for the future": when a similar task comes along, we won't start from scratch.
In a month and a half - zero reuses. Not one file was needed a second time. I archived the layer.
This is the same swarm, just frozen into files. The logic is identical: since we don't know what we'll need, let's make plenty, it might come in handy. The difference is that extra agents burn budget immediately, while dead artifacts burn attention gradually. They're visible in the file tree, they end up in the context, you have to account for them when refactoring and explain them to a new person.
The check is simple and unpleasant: any artifact created "for the future" should have a creation date and an access counter. If the counter is at zero after a month, that's not groundwork, that's ballast.
flowchart TD
T["Задача"] --> Q["Триаж: 5 вопросов"]
Q --> S{"Стоп-критерий сформулирован?"}
S -- "нет" --> X["Не начинаем"]
S -- "да" --> B["Бюджет: не более 5 вызовов"]
B --> E["Исполнитель - один, без роя"]
E --> K["Скептик - один проход, ищет отказ"]
K -- "не годится" --> E
K -- "годится" --> R["Обязательное ретро"]
R --> M["Память под контролем версий"]
M -.-> QThe rules that replaced quantity
I introduced them not in a lab but on live tasks - where the system works for a hotel chain and a northern tour operator, and where a wrong result costs not abstract reputation but someone's working day.
A call budget
No more than 5 model calls per task. One executor instead of a swarm. A budget is useful not because it saves money - though it does - but because it forces the system to choose. When you have five calls, you can't afford two for "think through the approach" and one for "clarify the wording." You have to get straight to work. The constraint acts as a design decision, not as accounting.
A five-question triage
Before launch, a task goes through five questions:
- what kind of task is this
- is it even worth doing
- what's the strategy
- what resources are needed
- when to stop
The second question cuts off half the work nobody asked for. The fifth is the main one. The stop criterion is formulated before the start, not once it's become clear that it isn't working out. If you can't formulate one, that's a signal: the task is badly posed, and the agent has nothing to do with it.
A retro after every task
Mandatory, not when the mood strikes. What the task was, what went wrong, what rule follows from it. That's how a system actually learns: not by gaining more roles, but by having something change in the rule set after every task.
Memory under version control
The system's organizational memory - the accumulated rules and conclusions - lives where the code lives, under version control (a tool that keeps the history of all file changes). Which means every rule has an author, a date, and a reason; you can see when it appeared; you can roll it back. Memory that lives in a database without history turns within a couple of months into a dump of statements where it's unclear who put them there and whether they're still true.
Verification: one skeptic instead of a crowd of judges
The popular verification scheme looks like this: several agents produce candidate answers, and a separate judge agent picks the best one. In research tasks, where you honestly don't know which strategy will work, it makes sense. In applied work I've dropped it.
There are three reasons. First, cost: N candidates plus a judge is N+1 calls where the whole task is allotted five. Second, the judge isn't an expert - it picks from what was brought to it. If all the candidates got it wrong the same way - and they tend to, because the model is the same and the task wording is shared - the judge will confidently pick the error and give it a high score. Third, diffuse responsibility: when five agents missed the error, there's nothing to fix, there's no mechanism to blame.
Instead - a single skeptic pass. The skeptic gets the finished result and one instruction: find the place where this is wrong. Not "rate the quality from one to ten," not "pick the best option," but show the specific place where it breaks.
The difference here is in how the question is posed, not in the number of checkers. A judge compares options against each other and will always find a winner - even among bad ones. A skeptic compares the result against reality and can say "not good enough" about the only option there is. That's the one check worth spending a call on.
Put it to work
What to check in practice
- Count your model calls over 24 hours and divide by the number of tasks solved. If the average surprises you, you don't know where the budget is going, and that's the first problem, not the money.
- Find every agent that runs on a schedule. For each one, ask: if a script with an ordinary condition made the decision, would the result change? Wherever the answer is "no," rewrite it as a script.
- Set a ceiling on calls per task. Start with 5 and watch what hits the limit: often it isn't the difficulty of the task, it's the chatter on the way in.
- Write out the five triage questions and run your last three tasks through them retroactively. Pay special attention to the fifth: was a stop criterion written down, or did the system stop because you switched it off?
- Go through the artifacts created "for the future" and count the accesses to each. Zero in a month - into the archive, not the backlog.
- Replace the "several candidates plus a judge" scheme with a single skeptic pass instructed to look for a specific failure. Run it on a dozen tasks and compare which scheme catches more real errors.
- Start a three-line retro after every task and put it under version control next to the code.
How to tell it worked
There's one main criterion: spending has stopped being a surprise. You can state the upper bound on calls before launch, instead of learning it from the billing page.
The other signs:
- every task has a written stop criterion, and you can show it as text rather than recall it from memory
- over time the number of rules grows, not the number of agents; if it's the other way around, you're still treating the symptom
- an incident review ends with a change to a rule, not with a new agent
- there are no artifacts in the repository that nobody has touched in a month
- the skeptic at least sometimes says "not good enough," and that holds up in practice. A skeptic that always agrees is decoration, and its call should be thrown out.
The reliability of an agent system comes not from the number of executors but from discipline: stop criteria, a call budget, learning through retros, and memory under version control. A swarm looks like strength and behaves like expense. Five rules cost less than five agents - and unlike them, they work the same at any hour of the day.