The problem looked architectural. It was the order of the deploy steps.

The deploy runs the usual way: the script builds the images, brings up the containers, then runs the tests. The tests open the vector database - the same one the service that just started is already working with at that very second. After that it's luck of the draw: sometimes a test falls over, sometimes it's not the test that falls over.

I looked at this and saw an architectural problem. Two processes sharing one store directly - so the store has to be moved out. The right answer, the expensive answer. And, as it turned out, the answer to the wrong question.

What follows is the breakdown: how this was actually put together, why the obvious solution was overkill, what we changed instead, and - most importantly - how to tell the case where changing the order of operations is enough from the case where you genuinely can't avoid a separate service.

The situation: an embedded database and two processes

The system is two containers. A web service that answers requests. And a bot with a scheduler: it talks to people and also runs background jobs on a schedule. Both work with the same vector knowledge base.

A vector database is a store of chunks of text along with their numeric representations, which you search not by matching words but by closeness in meaning. But for this story, what matters more is how it's connected.

And it's connected in embedded mode. That means: the database isn't a separate program you talk to over the network, it's a directory of files on disk. A library inside the process opens that directory and works with it directly, the way an application works with a file. There's no server in the middle.

Both processes open the same directory. That's exactly the arrangement any architecture review flags in red.

flowchart LR
  P1["Процесс: веб-сервис"] --> D[("Каталог с файлами базы")]
  P2["Процесс: бот"] --> D
  P3["Тесты при выкладке"] -.->|"третий, посторонний"| D
  N["Сетевого сервиса базы нет"] -.- D

Why the architectural solution suggested itself

The textbook answers this without hesitating: stand the database up as a separate service and talk to it over the network. One owner of the data, everyone else a client, concurrent access sorted out by the database itself. Nothing to argue with, it's the right solution.

The right solution had a price tag:

• separate work - rewrite the connection, migrate the data, make sure search answers the way it used to
• a separate container - you have to build it, update it, watch it
• a separate point of failure - the database service goes down and both processes go down with it at once, where before they failed independently of each other.

And this is where it's worth stopping and asking: what problem, exactly, am I closing with these three items? The answer was unpleasant: a hypothetical one. Two live processes reading one database could theoretically collide - but they weren't colliding.

And the thing that was actually breaking was breaking somewhere else. There was one real problem, and it was already here: during deploy, the tests opened the same database on top of a live process. Not "two services competing under load," but "during the deploy a third party climbs into the directory." A network service would have fixed that too - incidentally, as a side effect, at the cost of everything listed above.

flowchart TB
  subgraph WAS["Было"]
    W1["Поднять сервис и бота"] --> W2["Прогнать тесты"] --> W3["Тесты открывают базу<br/>поверх живых процессов"]
  end
  subgraph NOW["Стало"]
    N1["Тесты в одноразовом<br/>контейнере"] --> N2["Контейнер погашен"] --> N3["Поднять сервис и бота"]
  end
  WAS ==> NOW

What we did instead

We changed the order in the deploy script.

Before: bring up the working processes, then run the tests. The tests started at the moment when the service and the bot were already holding the database open.

Now: the tests run in a separate throwaway container - before the working processes come up. The container lives exactly as long as the tests take, then dies. If the tests don't pass, the working processes simply don't come up.

The result: nobody opens the database on top of a live process. Not because we forbade it with some mechanism, but because the moment when that was possible no longer arrives.

Embedded mode stayed as it was. We didn't touch it at all - not one line in the connection code. Only the deploy script changed.

It still strikes me as important that the fix turned out to be in a different layer from the one that hurt at first glance. The symptom looked like "storage architecture," and the cause was sitting in "the order of the deploy steps."

When you can do this, and when you can't

An honest caveat, without which this trick turns into bad advice.

It works because:

• the load is small - the processes aren't constantly jostling each other in the database
• writes are rare: reindexing runs on a schedule, not continuously. Reading at the same time is harmless. What's dangerous is someone writing while another is reading or writing.

The moment writes become heavy and parallel, the trick stops working. No ordering of steps in a script will save you from two processes rebuilding the index at the same time: they share a directory on disk, and they have nowhere to come to terms. That's when a network service stops being "right by the textbook" and becomes mandatory - because you need exactly one owner of the data who puts writes in a queue.

Signs you're still in the safe zone:

• there's exactly one writer, it writes on a schedule, and you know that schedule
• everyone else only reads
• you can name, one by one, every process that opens the directory.

Signs it's time to move the database into a separate service:

• there's more than one writer, or a write is triggered by a user event
• you're about to scale processes by running copies - a second copy of the service means a second holder of the directory
• you've already hit a corrupted index and fixed it by reindexing.

A cheap operational solution isn't "forever." It's "as long as the conditions hold." The conditions are worth writing down next to the deploy script, so that a year from now someone - possibly you - understands what exactly the whole thing rests on.

A separate lesson: the documentation described something that didn't exist

While I was digging into this, a detail surfaced. The system description at one point listed a separate database container. It had never existed. Not "it was there, we removed it, and forgot to update the docs" - it was never there in the first place.

The conclusion is simple and boring: check the code and the list of running processes, not the description. A description is somebody's version of the intentions at some point in time. It doesn't take part in the deploy, it doesn't open anything and doesn't block anything. If I had trusted it, I'd have been hunting a race between two services and a network store that doesn't exist, and I'd have found nothing.