Confabulated Numbers in Generated Text, and the Gate That Stops Them

In my content pipeline, the model was writing a short note: one provider had cut its API prices. I read the draft and couldn't find anything to object to. The price of the main model matched the price list. The paragraph held together, the conclusion followed.

The invented part was the numbers next to it. For two neighboring models from the same provider, the model quoted output at $8 instead of 12 and input at 0.50 instead of 0.20. This isn't a hallucination about a product that doesn't exist, not an invented company, not a link to nowhere. Four figures that appeared in no source, sitting exactly where a reader expects a figure.

What follows is why errors like this don't get caught by eye, how I built a mechanical check of numbers against a facts file, and where that check honestly doesn't work. By the end you'll be able to assemble your own gate (an automated check that either lets material through or stops it) that blocks publication on any number that isn't in your facts file.

The model lies confidently and in the details

The dangerous thing isn't the large invention. A large one is visible: if the model writes that input costs 400 dollars per million tokens, you trip over it on the first read, because the number falls outside the familiar range. The error that makes it to publication looks different.

In my case the invented numbers were the same order of magnitude as the real ones, in the same currency, in the same notation, in the same sentence position. The model didn't hesitate and didn't qualify anything. Generated text has no difference in tone between "I took this from the price list" and "I extrapolated this from the shape of the numbers around it" - on the page, both sentences read the same.

This isn't a one-off quirk of one particular model. I'd used the same technique earlier in a different pipeline, where the model produced absorption percentages for a substance that weren't in the source documents. Different domain, different documents, the same handwriting: a plausible number of the right magnitude in the spot where a number is required.

Why human proofreading doesn't catch it

To catch this by eye, you have to open the provider's price list and check every line of the draft against every line of the price list. Not skim it - actually check: find the model, find the column, match the value.

On the first note, you'll do it. On the second, you'll do it faster. On the third in a row, nobody does it, myself included. A person reading text checks coherence: whether the conclusion contradicts the premise, whether the transition between paragraphs broke, whether an idea repeats. A number that's consistent in format with its neighbors and doesn't break the paragraph's logic passes proofreading, because proofreading fundamentally isn't about that.

There's another reason, a less pleasant one. You're reading text you commissioned yourself - and you read it as the author, not as a skeptic. You already know what the note is supposed to say, and your eye confirms the expectation instead of testing it. Attention is a poor instrument against an error that disguises itself as normal. It runs out sooner than the flow of material does.

flowchart LR
  A[Файл фактуры: выверенные числа] --> B[Генерация черновика]
  B --> C[Извлечение сумм и процентов из текста]
  A --> D{Число есть в фактуре?}
  C --> D
  D -- да --> E[Публикация]
  D -- нет --> F[Стоп: разбор вручную]

The mechanical check: numbers against the facts file

The conclusion I reached: numbers should be checked not by a person but by a comparison of sets. The technique has two parts, and both are mandatory.

Part one - the facts file. Before generation, I write out into a separate file every verified number in my own words, each with a note on what the number is: which model it belongs to, whether it's input or output, what the charge is for. Not a copy-paste of the page, but a manual extract on the topic of the piece.

The manual extract isn't a formality here. While you're transcribing a number from the price list, you verify it once - and that's the only moment when verification is cheap. The note beside the number matters later: when the gate stops publication, you need to understand in one second which number it failed to find.

Part two - the script. It pulls every monetary amount and percentage out of the finished text and checks them against the list of numbers from the facts file. Any number that isn't in the facts file stops publication. It doesn't flag, doesn't warn, doesn't write to a log - it stops.

It's essential that the decision isn't made by a model. No "ask a second model to check the first": this is a comparison of a finite set of numbers from the text with a finite set of numbers from a file, in ordinary code, with a deterministic result. In that story about API prices, the gate would have stopped the note on the very first invented value: 8 and 0.50 simply aren't in the facts file, which says 12 and 0.20.

import re, sys

# денежные суммы и проценты: $0.20, 12 долларов, 0,50 долл., 8%
PATTERN = re.compile(
    r"\$\s*\d+(?:[.,]\d+)?"
    r"|\d+(?:[.,]\d+)?\s*(?:%|долл\w*|центов?|рубл\w*)"
)

def values(text):
    found = set()
    for m in PATTERN.finditer(text):
        raw = re.search(r"\d+(?:[.,]\d+)?", m.group()).group()
        found.add(float(raw.replace(",", ".")))
    return found

facts = values(open("facts.md").read()) # только выверенные числа по теме
draft = values(open("draft.html").read())

unknown = sorted(draft - facts)
if unknown:
    print("чисел нет в фактуре:", unknown)
    sys.exit(1) # публикация не состоится

Where the technique fails and what to do about it

I found two limitations in practice, and both are worth knowing before you start trusting a green status.

A raw dump kills the check. The temptation is obvious: copy the whole table into the facts file so it's "definitely enough." After that, the check stops working. A large dump contains nearly any number, and an invented value coincidentally matches somebody else's row - the gate finds a match and lets the material through. Hence the rule I now hold to strictly: the facts file contains only verified numbers on the topic of the piece, not everything you managed to copy. The broader the facts file, the weaker the gate.

Checking by value doesn't catch swapped numbers. It verifies that a number is present in the list, not what it's attached to. If the model takes the real price of one model and assigns it to another, both numbers will be in the facts file, the sets will match, and the gate will stay silent. This class of error stays with the human and the editor - and I prefer to say so directly rather than pretend automation has closed the subject.

What to do about it: treat the gate not as a guarantee of truth but as a way to remove one specific class of errors from a person's plate entirely. You no longer read invented numbers at all. The remaining work - checking that a number is attached to the right object - you still do by hand, but now it's the only thing you do by hand, and your attention stretches that far.