Context Windows: Why AI Forgets Things Mid-Conversation

You are working through a long contract with an AI assistant. At the start you said, in plain words: "Recheck every date against the signing date." Forty minutes later, the revised draft it returns has lost that requirement without a trace. Your first guess might be that it got dumber, or stopped caring. Neither — it simply could no longer hold what you said at the beginning.

That is the context window at work. Once you understand it, your expectations of an AI's "memory" change completely.

The window is a workbench, not an archive room

The context window is the ceiling on what a model can hold in one conversation: every message you send, every reply it writes, every document you paste — all spread out on one bench. The bench has a fixed area, measured in tokens (for what a token is, see What Are LLM Tokens, and Why Does AI Charge by Them?), and different models have benches of different sizes. While there is room, all is well; once the bench is full, something old must come off before anything new goes on. Note that the bench holds more than the conversation you can see: the system prompt and the results returned by tools take up area too.

Here is the part that matters: the model has no memory beyond the bench. It is not a forgetful person — it is a worker with a bench and no drawers. A person who forgets can still dig out their notes; content pushed off the bench simply ceases to exist for the model — it does not even get the nagging sense of having forgotten something.

Why it "forgets" mid-conversation

When a conversation outgrows the window, systems generally do one of two things: cut the earliest content off, or compress the older parts into a rough summary. Either way, detail drains away — and the first casualties are usually the requirements, definitions and background you laid out at the start. Hence the classic pattern: perfectly in sync in the first half, oddly off-target in the second. The model did not change; the bench holds a different set of things now.

A related confusion follows from this: open a new conversation and it knows nothing about the previous one — the bench has been cleared and reset. Some products do "remember" your preferences across sessions, but that is the product keeping notes outside the model and laying them back on the bench at the start. The model itself has not grown a memory.

Benches keep getting bigger — problem solved?

In recent years model windows have grown from a few pages' worth to a few books' worth, and that genuinely helps: tasks that once could not fit now do. But two new problems arrive with the size. First, fitting is not the same as holding: when content gets very long, the model pays visibly less attention to the middle than to the beginning and the end, so the middle of a long document is where things quietly slip — like an overloaded bench where the items along the edges stay visible and the ones buried in the centre get missed. Second, cost: every block on the bench is billed, and re-billed each turn, so the fuller the bench, the more expensive every round. A several-fold larger window is good news, then — but it neither removes the need to curate what goes on the bench, nor turns long conversations into a free lunch.

Three habits you can adopt today

Keep the bench in mind and the habits follow naturally:

• State important requirements at the very start of the conversation — and halfway through a long session, restate the key constraints instead of assuming they are still on the bench.
• Never paste in an entire manual. Let the system retrieve and hand over only the passages each question needs — cheaper and more accurate, and exactly the idea behind retrieval-augmented generation (RAG).
• For long working sessions, clear the bench on purpose: ask for a summary of decisions so far, then open a fresh conversation carrying just that summary.

What the three habits share is treating the bench as a scarce resource: keep the critical content on it at all times, and keep the unnecessary off it altogether.

When buying, don't let the window number steer

Window size is one of the loudest numbers in model marketing today, but for business procurement it is an entry ticket, not a scorecard. The questions worth asking: how big a bench does a single task in your scenario actually need? Can the model hold on to the middle of very long input? At equal quality, whose bench is used more economically? A solution that keeps each call's input small through retrieval often beats one that relies on heroically stuffing a giant window — on both cost and stability. Put differently, the window decides what can fit; it does not decide how well it is used — that depends on how you organise what goes onto the bench. For a decision-maker, the window number is like engine displacement: worth a glance, but nobody buys the car on displacement alone.