Roles, Not Rankings: How I Choose Models for a Pipeline

It almost always starts the same way. Someone puts together their first working pipeline - a chain of steps where text is passed from one model call to the next - opens the latest language model leaderboard, takes the top entry, and plugs it into every step in a row. The system works, and that feels good. Then the bill arrives, and it turns out the strongest and most expensive model in the system spent most of its time going through lists and throwing out what didn't fit.

The usual next move is to swap the expensive model for a cheap one and relax. That's a mistake too, just a quieter one.

I'll show how I choose models in my own setup: not "the best one", but different ones - for roles inside a single system. You'll walk away with three parameters the choice actually runs on, a trap I fell into myself, and two technical details that make the system fail somewhere other than where you'll go looking to fix it.

Why "which model is best" is the wrong question

A place in the ranking is an averaged result on a set of benchmarks: math, code, reasoning, knowledge, sometimes all at once. Averaging over many different tasks is exactly what you don't need. You don't have "tasks in general", you have a specific step of a specific pipeline: pick a few suitable items out of a long list, translate a paragraph, find the right chunk in a knowledge base, put together coherent text.

A model in a system isn't an employee you hire because they're the smartest on the market. It's a part in a particular position, and the position has requirements. The top line of the ranking answers none of them.

The right question sounds like this: which model does this step need, and why. And the answer is almost never the same for the whole system.

flowchart LR
  A[Поток входящих материалов] --> B[Роль: массовое чтение и первичный отбор]
  B -->|быстрая дешевая модель| C[Короткий список]
  C --> D[Роль: рассуждение, редактура, перевод]
  D -->|сильная модель| E[Готовый результат]
  F[Отраслевая база знаний на русском] --> G[Роль: поиск по смыслу]
  G -->|модель отечественного провайдера, своя квота| D

Roles, not rankings: how it works in my setup

My system has several roles, and each one has its own model.

• Bulk reading and first-pass selection of material - a fast, cheap model. It goes through the incoming stream, looks at each item against formal criteria, and leaves a short list.
• Reasoning, editing, and translation - a strong model. It comes in later and works not with the stream but with the little that made it through the selection.
• Working with a Russian-language industry knowledge base - a model from a domestic provider with its own quota. Quota here means a separate limit and a separate balance, not shared with the other calls.

Splitting by role gives you the main thing: the expensive model gets called rarely and for good reason, not for every little thing. Notice where the savings come from. Not from my having found a cheaper model, but from the expensive call no longer happening where it wasn't needed. Those are different things, and the second one works more reliably.

The three parameters the choice actually runs on

First: language and domain

To search by meaning rather than by word matching, text is turned into a vector - a long set of numbers where fragments close in meaning end up near each other. This is called vectorization, and a separate model does it. That's how knowledge base search works: the question gets vectorized, and the system pulls out the document chunks nearest to it.

For a Russian-language knowledge base, the quality of vectorization in Russian matters more than the model's place in the general ranking. A model can be considered perfectly strong in general and still be worse at telling apart the shades of phrasing of your language and your industry. You have to check that on your own texts and your own real questions, not from a table.

Second: the nature of the task

Selecting from a list by criteria and generating coherent text are different tasks, and they load the model differently. A fast model selects well and writes long texts badly. In my case it looked like this: on a long text it slides into an impersonal brochure and starts inventing facts - that is, confidently producing plausible details that weren't in the source.

So the conclusion "the model is weak" is wrong. The right conclusion: this model is good at selection and no good at generation. It stays in the system, just in its own place.

Third: how you pay

The usual scheme is per-unit payment for tokens. A token is a piece of text, roughly part of a word; you pay both for what you sent the model and for what it wrote back. Under this scheme cost grows along with volume, linearly and without a ceiling.

Part of the work is worth moving off per-unit payment onto a subscription with a fixed price. For a stream of same-type tasks this changes the economics radically: you pay for the month, not for every run, and growth in volume stops being growth in the bill. One-off and rare tasks, on the contrary, live perfectly fine on per-unit payment.

flowchart TB
  A[Регулярный вызов, который на самом деле не нужен] --> B[Шаг 1: перевести на модель дешевле]
  A --> C[Шаг 1: проверить, нужен ли вызов вообще]
  B --> D[Расход по вызову меньше в 7 раз]
  C --> E[Вызов отключен, расход по нему исчез]
  D --> F[Силы потрачены на оптимизацию мусора]
  E --> G[Сначала нужность, потом цена]

The cost-cutting trap

Now about the mistake I tested on myself.

I had a recurring call that cost noticeable money. I moved it to a cheaper model and got a sevenfold saving. A nice number. The problem is that this call wasn't needed at all - it simply should have been turned off. Instead of a sevenfold saving I could have had a complete one, and I spent effort optimizing something whose place was in the trash.

Hence the order of operations I now stick to strictly: first check whether the call is needed, then what it costs. Making it cheaper is the second step, not the first. It's very tempting, because it gives a fast, measurable result and doesn't require thinking about what's going on in the system.

The token ceiling you never spend

When you call a model you set an upper bound on the length of the response - the parameter usually called max_tokens. That isn't spending, it's a declaration: "the answer may be this long".

Providers sometimes check ability to pay against the requested ceiling rather than the actual spend. A task that asks for the model's maximum ceiling for the sake of a ninety-token answer starts failing when the balance runs low - even though it spends pennies. The symptom is deceptive: spend is small, but the calls don't go through. The cure is to set the ceiling by the real length of the answer, not "bigger, just in case".

One key for everyone

Several agents can share one access key and one balance. When the balance runs out, they all fail at once, but it looks like one of them broke - the one whose error you noticed first. You go off to fix its logic, while the cause is shared and external.

The sign of exactly this situation: several different failures in different places within the same minute. It helps to know in advance which agent goes through which key, and to split critical roles across different balances.