One day I opened the channel and saw my own article there, with tag names sitting in the text literally - angle brackets in the middle of a sentence. Next to it was a link to the original source, broken apart into half an address and a fragment of a word. The text had been proofread beforehand. It was fine. It just didn't reach the reader in the shape I sent it.
This isn't a story about a bad model. It's a story about the fact that some requirements for a text are pointless to ask for - they have to be checked. Below I'll go through which requirements are worth moving into code, why markup breaks more quietly than anything else, and how to structure the error you send back so the next attempt fixes exactly what's broken instead of rewriting everything from scratch.
A prompt is a request, not a guarantee
A prompt is the text instruction you give the model before generation: write like this, don't write like that, quotes like these, links like those. It's a convenient place, rules go in there easily, and for a while it seems like that's enough.
In my experience, a rule written into the prompt is followed almost always.
That "almost" is what costs the most. Not because there are many violations - there are few. But because the moment of discovery is chosen as badly as possible: the violation surfaces at the reader. Ninety-nine texts in a row come out exactly as you asked, you stop looking at that detail - and the hundredth comes out different.
The difference between a request and a guarantee is who answers for the outcome. A request is carried out by the model, and the carrying out is probabilistic. A guarantee is given by code: the check either passed or it didn't, there's no third state. The prompt doesn't go anywhere in the process - it stays the way you get a good text on the first try. It just stops being the only place where the requirement lives.
What makes sense to check with code
The selection criterion is simple: what goes into code is whatever you can answer yes or no to without reading the text. Not "is it written interestingly," but "is the thing that should be here actually here."
In my content pipeline, this is what gets checked by code:
- is there a link to the original source - not "is it appropriate," just is it there
- is the link formatted as words rather than a bare address
- are there any bare addresses left anywhere else in the text
- does the markup have any emphasis, or did the text arrive as one solid grey mass
- did straight quotes slip in instead of guillemets
- does the length fit within the given bounds
- are there any clichés from the blacklist.
There's nothing clever in these checks, and that's their value. Each one takes a few lines, runs instantly, and doesn't depend on the mood of the generation. The list looks petty right up until it catches a text that was otherwise ready to publish.
This is also where the boundary runs that I try not to blur: code checks form, a human checks meaning. A mechanical gate doesn't replace proofreading. It takes the drudgery out of it - the part where you hunt for a straight quote with your eyes and count whether the text has even one emphasized fragment. After the gate, proofreading does what it exists for: are the claims correct, does the structure work, did the text lie in some detail.
flowchart TB
D["Черновик от модели"] --> G{"Механические проверки"}
G --> G1["Разметка сбалансирована"]
G --> G2["Цифры есть в фактуре"]
G --> G3["Ссылки оформлены словами"]
G --> G4["Имен заказчиков нет"]
G1 & G2 & G3 & G4 -->|"хоть одна не прошла"| BACK["Точный список замечаний<br/>автору: правка, а не<br/>новая генерация"]
G1 & G2 & G3 & G4 -->|"все прошли"| HUMAN["Вычитка о смысле:<br/>верно ли, работает ли<br/>структура"]Markup: why it breaks the most quietly
Markup is a separate class of problem, and a nastier one than the rest.
The messenger the text goes out to accepts only a few tags. If the markup is unbalanced - you opened emphasis and didn't close it, nested one tag inside another the wrong way - it doesn't display the text crookedly. It rejects the message entirely. Which means an error in a single character turns into "there was no publication," and you find out about it only from the absence of the post.
So markup in my setup is handled in two passes. First the script fixes what can be fixed mechanically: line breaks, lists - things where the correct version follows unambiguously, no guessing. And what can't be fixed mechanically, it doesn't try to guess: it rejects it and sends it back to the author.
Now about that failure from the beginning of the article. The text went to the channel with literal tags and a broken link because the publishing layer was escaping the markup - that is, turning the special characters into ordinary ones - while the pipeline puts that markup there deliberately. Two layers, each right in its own way, and the result is visible only at the output, at the reader.
After fixing it I added a safety catch: if escaped markup is left in the finished message, publication doesn't happen at all. Not "publish and fix," not "publish with a caveat" - just don't publish.
The principle behind it: not publishing is cheaper than fixing what's published. An unsent post costs you ten minutes and mild irritation. A sent and broken one costs you an edit in front of your audience - and in a channel where messages can't be quietly rewritten, a deletion and a repeat send on top of that.
How an error should come back to the author
A gate that can only say "didn't pass" is nearly useless. It reports that something somewhere is wrong and leaves you to find it yourself - that is, it hands back exactly the work you wrote it to be rid of.
The error should come back as a list of specific complaints: there's no link to the original source here, there are straight quotes in three places here, the text is below the lower bound here, there's a cliché from the list here. Then the next attempt fixes exactly what needs fixing and doesn't touch the rest.
This matters both when you're doing the fixing yourself and when the model is rewriting the text. A blanket rejection provokes a rewrite from scratch - and along with the problem spot, the good parts that were already done get carried away too. A precise list turns the second pass into an edit rather than a new generation.
# Кириллица только с заглавной буквы: иначе короткое имя совпадет с обычным словом
CLIENT_RE = re.compile("|".join(CLIENT_NAMES))
def client_names(text):
"""Имена заказчиков, просочившиеся в публичный текст."""
return sorted({m.group(0) for m in CLIENT_RE.finditer(text or "")})
# в проверке статьи:
found = client_names(text)
if found:
errors.append("имена заказчиков в тексте: " + ", ".join(found))Put it to work
What to check in practice
If you want to set up a gate like this yourself, start small - there's no need to write a big set of checks right away.
- Write out the rules that currently live only in the prompt. Just as a list, as they are.
- Mark the ones you can answer yes or no to without getting into the meaning. Those are your candidates for code.
- Take one such item - bare addresses in the text, say - and write a check a few lines long. Run it over a dozen already-published texts; if you find a violation, you've just proved to yourself why "almost always" isn't enough.
- Separate the errors that can be fixed mechanically from the ones that can't. Fix the first kind silently, reject the second.
- Word the rejection text so that it enumerates specific complaints rather than summarizing.
- If there's another layer between generation and publication, check what it does to your markup. It's worth verifying that the two layers' expectations match once, explicitly, instead of finding out on a live post.
- Where a broken result costs more than a missing one, put in a safety catch that refuses to publish.
How to tell it worked
The main sign is that you stop proofreading the text for form. Not because you've gotten lax, but because the form has already been checked: if something's wrong, the text simply won't reach proofreading. Attention frees up for meaning, and it shows in the kind of comments you now write on drafts.
The second sign is that the gate's rejections stop surprising you. You read the list of complaints and immediately know what to fix; no time goes into working out "what was even meant here."
The third is that the number of post-publication corrections tends to zero. Not halves - actually goes to zero for the class of problems you closed with code.
And a side effect I didn't appreciate right away: the list of checks acquires a history. Every check in it is the trace of a specific breakage, not the result of musing about quality in general. Rereading the list, you can see what the system learned on and at what cost. It's the most honest process documentation I've ever had: impossible to write in advance and pointless to prettify.