The demo lasts fifteen minutes. You ask the bot three questions - it answers in a second. You show the report mailout - it goes out on time. Everyone is happy, the work is accepted. A week into operation a message arrives: "the bot is silent." Not "it threw an error," not "it answered incorrectly," but precisely silent.
I was building a Telegram bot for hotel automation: it answers staff questions from an internal knowledge base and sends out reports on a schedule. In the demo it worked. In production it broke in three different ways - and not one of them can be reproduced in a demo, in principle. Not because I demoed it badly, but because a demo is arranged so that these failures are impossible inside it.
What follows: three concrete classes of failure, the mechanics of each, and what should have been written into the code ahead of time.
The demo is the best case, production is everything else
A demo always has three conditions lined up.
- One user. That's you, and you ask your questions one at a time.
- A fresh process. The program was started a minute ago, nothing has had time to pile up, hang, or drift out of sync.
- A good day at the provider. The external service you're asking for an answer - in my case a language model, that is, the very AI that phrases the text - responds quickly.
Production is all the other combinations. Users arrive at the same time. The process lives for days and survives everything that happened over those days. The provider sometimes doesn't answer at all - and that's a separate state, not the same as an error: an error you'll see and handle, whereas silence looks like "still working on it."
That's why a demo checks exactly one scenario - the happy path. And the system breaks on all the others.
flowchart TB
U["Запрос пользователя"] --> C["Вызов модели"]
C --> P{"Провайдер отвечает?"}
P -->|"да"| OK["Ответ пользователю"]
P -->|"ошибка"| E["Ошибка видна,<br/>обработана"]
P -->|"молчит"| W1["Без таймаута:<br/>ожидание не кончается,<br/>бот не отвечает никому"]
P -->|"молчит"| W2["С таймаутом 60 с:<br/>сдаемся сами,<br/>отвечаем пользователю"]Failure one: a call with no timeout hangs forever
A timeout is a limit on how long you wait: "I'll wait for an answer no longer than N seconds, then give up on my own." If you don't set it explicitly, in many libraries the wait is bounded by nothing at all.
That's exactly how it was for me: the language model call was made without a timeout. In the demo this is invisible - the answer comes back in seconds. In production the provider sometimes didn't answer at all, and the request hung indefinitely.
Now the detail that beginners usually miss. The bad part isn't that the user didn't get an answer. The bad part is that nobody found out anything. There's no error - which means no log entry, no "please try again" message, no retry. The task just sits there and takes up process resources. From the outside this looks like "the bot is silent," and from that description you can't work out the cause.
The cure is simple: an explicit timeout on every network call. Ours is 60 seconds. The number isn't magic: it's about how long a person is willing to wait for an answer in a chat, and yours may be different. What matters is that it exists at all - then "hangs" turns into "failed with a clear error," and with an error you can actually do something: log it, show it, retry it.
The same thing in the deploy script
The same defect turned up where you don't expect it. The deploy script (rolling out a new version to the server), after starting the service, polled it: "are you up?" - and waited for an answer with no time limit. If the service didn't answer, the script itself hung: the rollout ended in neither success nor failure. A 5 second limit per attempt was added.
The rule is the same in both cases: a network call is not a function call. It may never return.
Failure two: a heavy warm-up blocks startup
Warm-up is preparing the heavy parts of the system at startup, so you don't pay for it on every request later. At startup my bot brought up the search pipeline: the mechanism that, given a staff member's question, finds the relevant pieces of the knowledge base and hands them to the model. That requires loading the model and opening the database - not a fast operation, which is why it's done once.
The mistake wasn't in the warm-up itself, but in where it sat. It was placed before the scheduler started (the component that runs tasks on a schedule - in our case the report mailout) and before message handling began. Which means that until the warm-up finished, the bot wasn't alive in any sense.
While everything is fine, you can't see the difference: the warm-up took half a minute, nobody noticed. But if the warm-up stalled - and it depends on disk, memory and external services - the bot didn't answer at all and the schedule didn't work. One slow operation at startup switched off both functions at once.
The cure: the warm-up was moved into a background task after startup. The bot accepts messages immediately, the scheduler starts immediately, the heavy part loads in parallel. This adds a question you'll have to answer deliberately: what does the system do with a request that arrives before the warm-up has finished. But that's an honest question with a clear answer - unlike "the service didn't start, cause unknown."
flowchart TB
subgraph WAS["Прогрев до старта"]
W1["Старт"] --> W2["Прогрев: модель и база"] --> W3["Завис - расписание<br/>не работает,<br/>бот не отвечает"]
end
subgraph NOW["Прогрев в фоне"]
N1["Старт"] --> N2["Расписание и прием<br/>сообщений"]
N1 --> N3["Прогрев параллельно"]
N3 --> N4["Вопрос: что делать<br/>с ранним запросом - <br/>честный и с ответом"]
end
WAS ==> NOWFailure three: a health check that takes everything down itself
A health check is a simple request that the service answers with "I'm alive." An external system pokes it regularly - monitoring, or whatever manages startup - and from the answer decides whether to consider the service working.
My check handler requested knowledge base statistics. The logic looked reasonable: check not just "the process exists," but "the data is in place, I really am ready to work."
The problem is that a health check has to answer in milliseconds and not depend on anything heavy. As soon as the database started to stall, the check stalled along with it. And here the coupling becomes dangerous: the instrument that is supposed to report the system's state turns out to be part of it and to share its problems. The external system sees silence and draws conclusions - and its conclusions are usually radical.
The cure: the check was reduced to the cheap question "is the database file there." No reaching inside the database, no statistics.
A distinction worth remembering: a health check answers the question "can I be asked things," not "is everything all right with the data." The second question matters too, but that's a job for monitoring and separate metrics, not for a handler on which the life of the service depends.
Put it to work
What to check in practice
You can check all of this on your own system in a single evening, without waiting for users.
- Find every place in the code where you go outside: to the language model, to the database, to someone else's API, to your own service from the deploy script. Write them out as a list.
- For each one, check whether a timeout is set explicitly. Not "there's surely one by default" - open the library's documentation and make sure. The absence of a default limit is more common than it seems.
- Look at what happens between process start and the first accepted request. Everything sitting in that stretch is a potential reason the service won't come up at all.
- Open the health check handler and look at what it reaches for. If it touches the database, the model or the network for more than "the file exists" - simplify it.
- Look over the deploy scripts with the same eye. They make network calls too, and they can hang too.
The way to check all of it at once is to swap the external service's address in a test environment for one that accepts the connection and doesn't answer. Not a non-existent one: that will honestly return an error right away, and what you need is precisely silence. If the system keeps accepting messages while that's happening, and the hung request fails after the set time with a clear error in the log - you did everything right.
How to tell that it worked
There's a single criterion, and it's stated without numbers: any external dependency can go silent, and that must not turn your service into a silent black box.
Break it down into three checkable statements.
- No request can wait longer than a set time - because every network call has a number in seconds that you can say out loud.
- The service accepts requests immediately after startup, regardless of whether the heavy part has had time to get ready.
- The health check answers equally fast both when everything is fine and when the database is slow.
The principle I arrived at after these three failures fits on one line: every external call gets a timeout, every heavy startup gets a background mode, every health check gets a cheap implementation that doesn't touch heavy subsystems.
And one last thing. None of the three failures was a logic error: the bot searched correctly, answered correctly, built the reports correctly. Everything broke at the boundaries - where the code stops being yours and starts depending on someone else's time. A demo doesn't show those boundaries, because fifteen minutes of a good day simply don't reach them.