I’m building a chat feature where users may keep one conversation going for days. Eventually, the history is too large to send with each request, and different models seem to hit the limit at different points. What’s a reliable way to decide what stays in the prompt without losing details the user expects the assistant to remember?
I’m especially unsure whether to summarize old messages, drop them, or store them somewhere and retrieve relevant parts later. A summary is cheaper to send, but it can blur exact names, numbers, or instructions. Retrieval sounds more precise, though I don’t want every new message to trigger a complicated search that returns irrelevant history.
What belongs in the next request?
It helps to treat the context window as a limited working space, not as permanent conversation storage. The current system and developer instructions, the latest user request, and the recent turns usually deserve priority. Older details can be handled separately depending on whether they are still useful.
- Keep recent turns intact when their exact wording matters to the current exchange. Preserve complete user-assistant turns rather than cutting a conversation in the middle of a reply.
- Summarize older discussion when it contains durable context, such as the user’s project, preferences, or decisions already made. Keep concrete values and names explicit, and mark uncertainty instead of guessing.
- Retrieve older details when the conversation has a large archive and only a small portion is likely to matter. Search results should be treated as supporting context, not as a replacement for the latest instructions.
- Discard low-value material such as repeated greetings, stale tool output, or details that have no bearing on the current task.
These approaches can be combined. For example, an application might keep a short rolling conversation summary, retain the last few turns verbatim, and search older messages when the current request refers to something from the past. That gives the model a compact picture without pretending every message is equally important.
Budget for more than the user’s messages
The input is not just the visible chat transcript. System instructions, tool descriptions, retrieved records, and the model’s reserved output all consume space. I’d leave room for the answer and avoid building a request right up to the published context limit; a long tool result or an unexpected message can push it over.
Token counts also vary. Providers may use different tokenizers, and model or API formatting can add overhead that is not obvious from the plain text. If the application supports several models, estimate against the selected model where possible, then keep a safety margin. Character counts alone are a rough warning, not a dependable context check.
Summaries need maintenance
A summary can become misleading if the application keeps appending new information without reconciling contradictions. If a user changes a preference, the stored version should be updated rather than leaving both the old and new statements as equally current. It is also useful to separate confirmed facts from temporary plans: “uses PostgreSQL” is different from “considering a move to PostgreSQL.”
For exact details—an invoice number, a quoted sentence, or a configuration value—don’t assume a compressed summary will preserve them. Store those details in a structured record or retrieve the original message when needed. Retrieval-based context works best when the application can identify what the user is referring to and return a small, relevant excerpt, not a pile of loosely related history.
One practical question for anyone implementing conversation history management: how are you deciding which older details are durable enough to summarize, and which need to remain searchable as original messages? I’m interested in approaches that work across OpenAI, Claude, Grok, Perplexity, and DeepSeek without relying on one provider’s context-window behavior.
