Five Months in One Chat: Where a Chat Model Helps and Where It Starts to Fail

On May 18, 2026 I spent half an hour proving to my tutor that the models I work with every day exist. The tutor was Gemini, the argument was about Claude Sonnet 4.5 and Opus 4.6, and the model explained to me three times that Anthropic has no such thing and that my brain was «throwing in fours by association with GPT-4». I sent a screenshot of the documentation. The model answered: «I am reading the pixels strictly off your image» - and listed what it saw there. In its next answer, on the very same screenshot, it saw a different list.

The cost of that particular error was zero: I knew the right answer. But the tone was absolutely confident, complete with an explanation of why I was the one who was wrong. If it had been about something I don't verify every day, I would have believed it.

From April to August 2026 I went through an entire prompt engineering course (that's about how to phrase a task for a model so the result is reproducible rather than accidentally good) with a chat model in the role of tutor. I fed it the lesson texts and the homework assignments, it gave me back a step-by-step plan and the text of the report. The whole conversation was saved, and what you can see in it is exactly what usually gets lost: not «AI helps you learn», but on which tasks specifically it helps and on which it starts getting in the way. By the end of this article you'll have the line between those tasks and three checks that catch the standard failures.

The experiment that happened by accident: five months of studying in one chat

I wasn't planning an experiment. I just never started a new chat. By August it had accumulated 780 messages: 386 requests from me and 394 answers from the model, 326 pages if you export it to PDF.

The first thing that jumps out is the asymmetry in volume. The model wrote about 1.13 million characters, I wrote 152 thousand. 7.4 times more text. That's not a quality judgment, it's a description of the genre: I set direction in short lines, it unfolded them into prose.

The second thing is the structure of the requests. 78 out of 386 requests (every fifth one) are «write up the report for this homework». Writing report prose was the model's main job, not explaining the material at all. Another 60 requests are error text pasted in from an interface or a log, and they cluster in one place: the module on no-code tools, meaning systems where you assemble a scenario with the mouse out of blocks without writing code - n8n, Albato, Google Cloud.

The third thing is more interesting than the first two. 38 requests contain a direct complaint aimed at the model: «you're making things up», «you're hallucinating», «stop confusing me». Those 38 are not spread evenly across five months. They fall at the end of the conversation - at the point where study tasks gave way to real projects. As long as I was solving a task from a lesson, one that has a known correct answer in the lesson text, the model was good. The moment a task no longer had a ready answer, the failures started.

What the tutor did well

I'll list them honestly, because this isn't an empty list.

• Turning lesson text into a step-by-step plan. I pasted in the lecture and the assignment, I got back a sequence of actions. The savings are real, especially when the material is written in a diffuse way.
• Report prose. Those 78 requests. A homework report is a genre with a predictable structure, where the content is already in my head and what's needed is the packaging. The model does this faster than me and no worse.
• Rephrasing what I didn't understand. Asking «explain the same thing, but simpler» costs one message and takes nobody's time but mine.

What all three have in common: the source data lies entirely inside the request. The model doesn't have to obtain, recall, or verify anything - only repackage what I gave it. On tasks like these, a chat with no file access remains an excellent tool, and no agent is needed here.

Failure one: outdated knowledge presented as fact

The argument about Claude Sonnet 4.5 and Opus 4.6 from the start of this article isn't an isolated case, it's a category. The model knows the world as of the moment of its training and doesn't distinguish «I don't know this» from «this doesn't exist».

In the same course it confidently told me about a model called GigaChat-Max-embeddings. No such model exists. To refute it, I needed four screenshots of the official documentation. It also calculated GigaChat pricing for me using OpenAI's rates and in dollars - that is, it took a familiar calculation template and plugged an unfamiliar name into it.

More dangerous than the error itself is the phrase «I am reading the pixels strictly off your image». That's a claim about its own capabilities, and it's false: the proof is in the text of the conversation itself, where two adjacent answers «read» different lists off one and the same image. The model isn't lying maliciously - it's constructing a plausible description of what ought to be on the screenshot and presenting it as an act of perception.

Failure two: invented results

This is the failure that made me stop trusting reports generated without pasted-in numbers.

I was running an A/B test - comparing two variants of a system on identical questions - on whether to include Chain-of-Thought in the prompt, the technique where you ask the model to reason out loud step by step before answering. The common belief: reasoning out loud improves answer quality.

In the report, the model wrote: «quality rose from 3.8 to 4.4».

Those numbers didn't exist. The actual measurement on 40 questions gave, on the Completeness metric (how complete the answer is), 3.55 versus 3.80 - minus 0.25. Hypothesis rejected. Meanwhile Hallucinations came out at 4.08 versus 4.28, meaning the control guard metric, the one that was supposed to catch a degradation along a different dimension, stayed in the normal range. The conclusion wasn't the one I expected: Chain-of-Thought reads to the model as permission to expand, not as an instruction to «check yourself twice». The resulting recommendation is to enable it conditionally, only when the retrieval score is high, that is, when the search actually found relevant chunks of documents and there's something to build on.

flowchart LR
  Q["A/B тест: включать ли Chain-of-Thought"] --> M["Что написала модель в отчете"]
  Q --> R["Что показал замер на 40 вопросах"]
  M --> M1["Completeness: рост с 3,8 до 4,4"]
  R --> R1["Completeness: 3,55 против 3,80, минус 0,25"]
  R --> R2["Hallucinations: 4,08 против 4,28, guard в норме"]
  M1 --> V["Таких замеров не проводилось"]
  R1 --> H["Гипотеза отклонена, CoT включать условно"]

Notice that the real result has more substance than the invented one. The model wrote a smooth «it got better» because that's how experiment reports usually end. Reality gave a negative result and a clear explanation of the mechanics - the thing the experiment was run for in the first place.

Failure three: no hands

The most expensive in time and the most boring. A model in a chat sees neither the project files nor my screen - it sees only what I copied into the window. And it honestly advises me on settings that don't exist in my interface.

The scenario: a bot in a no-code build was replying twice to a single message. About fifteen requests went into cycling through modes of the merge node - try this mode, now this one, now check this box. What helped was recreating the node from scratch. Meaning nothing that was suggested helped. A lesson budgeted at 20 minutes took 3 hours.

These are exactly those 60 requests with pasted-in error text. In a situation like that the model can't say «I don't know what's on your screen» - it generates a plausible list of hypotheses, and it sounds like diagnosis, though it's guess-and-check.

flowchart LR
  subgraph Chat["Чат без рук"]
    H1["Человек копирует фрагмент ошибки"] --> C1["Модель советует вслепую"]
    C1 --> E1["Перебор гипотез, настройки нет в интерфейсе"]
    E1 --> E2["20 минут превратились в 3 часа"]
  end
  subgraph Agent["Агент с доступом к файлам"]
    A1["Агент читает файлы проекта"] --> A2["Правка и проверка на месте"]
    A2 --> A3["Готовый проверенный результат"]
  end
  A3 --> R1["Чат оформляет отчет по вставленному результату"]

Where the line runs, and why it isn't about how hard the topic is

The turning point came twice, and neither time was because of a change of model.

The first was advice from my course mentor: don't dive into no-code, make a clean knowledge base with a proper document structure as the first stage. That's moving the work to where the result is checked with your eyes rather than guessed at through an interface.

The second was moving the work into an agent, that is, into a model that has tools: it reads the project files, edits them, and sees the result of its own action. After the move, the chat model's role shrank to one function: I paste the finished result into the chat and it formats a report out of it. Exactly the genre it was strong at.

The line doesn't run along how hard the topic is. Prompt engineering and evaluating RAG systems are hard topics, and the chat helped there. Setting a checkbox in a node is a primitive topic, and there it did damage for three hours. The line is this:

• All the data is in the request - the chat works. Rephrase, structure, format, explain.
• The data is outside the request - the chat needs hands, not hints. A file, a log, a screen, a real measurement, the current version of anything.

In the second case a model without tools doesn't go silent - it fills the gap with plausible text. That isn't a bug in one particular vendor, that's its job.

A side result of five months: the conversation turned out to be a ready-made dataset about the model itself. Every place where I irritably wrote «stop making things up» is free hallucination labeling, done in passing, with the context before and after preserved. If you keep long chats, you already have a dataset like this.