Tests Before Startup: Four Decisions That Keep a Broken Version Off the Server

I have a system of two containers: a web service and a Telegram bot with a scheduler. The code lives inside the image, not in a mounted directory - meaning the container doesn't look at a folder on the server where you could patch a file in place and restart the process. Because of that, any edit, even a one-liner, means rebuilding the image. There simply is no "fix it live" spot.

That sounds like an inconvenience, but it's actually a gift: since deployment is always the exact same procedure anyway, you can build that procedure once so that a broken version physically cannot reach users. By the end of this article you'll have that sequence of steps, and you'll understand why the fast path around the tests costs more than it looks like at the moment you choose it.

The temptation of the fast deploy

The fast path is a single command: rebuild and bring it up. It exists in almost every project, and it skips the tests entirely. The temptation is understandable: the edit is small, the full cycle takes longer, and "there's nothing here to break."

The problem is that a bot has no error page. A web service that failed to start is visible immediately - someone opens it and sees it. A bot that failed to start looks like silence: the user writes and gets no answer, and the scheduler just doesn't do its work at the appointed time. Nobody gets an error message, because there's no one to send it - the process never started.

So the fast path does exist for me, but it's not a working mode, it's an exception. It's there for an urgent patch when something is on fire right now. And it leaves a debt: afterwards the full cycle gets run, no exceptions. If you don't pay that debt, you end up with a server nobody knows the test status of - and the next deploy will be repairing two problems instead of one.

flowchart TB
  S1["1. Синхронизация файлов"] --> S2["2. Сборка образов"]
  S2 --> S3["3. Тесты в отдельном<br/>одноразовом контейнере"]
  S3 -->|"прошли"| S4["4. Подъем рабочих контейнеров"]
  S3 -->|"упали"| STOP["Стоп.<br/>Пользователи остаются<br/>на рабочей версии"]
  S4 --> H["Проверка после подъема:<br/>ловит «не поднялось»"]

The order that protects: tests before startup

The full cycle is four steps, strictly in this order:

• syncing files to the server
• building the images
• running the unit tests in a separate throwaway container
• bringing up the working containers

A throwaway container is a container that starts from the freshly built image, does one job, and dies without staying in service. That's exactly where the tests run: in the same environment as production, but without consequences.

The whole point is that step three comes before step four. If the tests fail, it never gets to startup. The previous version keeps running on the server - the one that worked yesterday. The user sees nothing, because as far as they're concerned nothing happened. This is fundamentally different behavior from "brought it up, saw the error, rolled back": between "brought it up" and "saw" there is always a window, and in that window the bot is silent.

Why before, and not after

There's a second reason, less obvious. My tests open the same knowledge base as the working process. When I ran them on top of already-running containers, a race appeared - the situation where two processes reach for the same resource at once and the result depends on who got there first. A test could fail not because the code was bad, but because at that moment the live bot hit the base.

A test that fails every other time is the worst kind of test: people stop believing it, and then stop looking at it. The "tests before startup" order removes the race without a single line of code: at the moment of the run, the working containers aren't up yet, so there's nothing to compete with.

The post-startup check and its limits

After startup, the script waits until the service reports that it's ready. There are two rules here, and I learned each of them somewhere other than the documentation.

The wait must be bounded in time. Without a bound, the deploy script once hung forever - it was waiting for an answer that was never coming. The worst part isn't that the version is broken, it's that the deploy ends in neither success nor failure, and you don't know what state the system is in. A bound turns a hang into an honest negative result, which is something you can act on.

The readiness check must be cheap. My handler, the one answering the question "are you alive?", went and fetched database statistics. As long as the database answered fast, everything was lovely. When the database stalled, the handler stalled with it, the wait ran into its time bound, and the deploy was falsely counted as failed - even though the service was up and running.

A readiness check answers exactly one question: the process started and is accepting requests. It does not answer the question "is everything fine in the system." The moment you start mixing substantive checks into it, it inherits every way those checks can break.

Hence its limits. The post-startup check catches "didn't come up." It doesn't catch "came up but calculates wrong" - that's the job of the tests that ran earlier. Don't try to cover with the second tool what the first one was supposed to catch.

flowchart LR
  R["Репозиторий"] -->|"едет"| SRV["Сервер"]
  subgraph KEEP["Остается на сервере"]
    K1["Секреты и .env"]
    K2["База данных"]
    K3["Журналы"]
    K4["Файлы, созданные<br/>пользователями"]
  end
  SRV --- KEEP
  DEL["Удаление лишнего<br/>без списка исключений"] -.->|"стирает"| KEEP

What must not travel with the code

A separate category of rakes is file sync with deletion of extras. That kind of sync brings the directory on the server to the state of the repository: it not only copies what's new, it erases everything the repository doesn't have.

One day it erased my working config with the target metrics. The file was needed by the system, but it wasn't in the repository - and from the sync's point of view it was garbage. Formally, everything worked correctly. In practice, I lost data that existed in exactly one copy.

The conclusion is simple: everything that lives only on the server must be in the exclusion list. At a minimum:

• environment files with secrets
• database directories
• logs
• user data

It's useful to keep this split in mind: the repository owns the code, the server owns the state. Code travels from the developer's machine to the server and can be overwritten without regret - it's reproducible from history. State is born on the server and isn't reproducible from anywhere. If a file is created or changed while the system runs, it isn't the repository's property, and the sync has no right to touch it.