When someone closes a tab or leaves a chat while an LLM is still streaming, should my server stop the generation too? I’m trying to work out the right behavior for an app that streams answers from a model API. At the moment, the browser stops displaying the response, but I’m not sure whether the upstream request keeps running—and whether I’m still paying for tokens nobody will see.
There seem to be two separate things to handle: stopping work that no longer has a viewer, and deciding what to do with any useful text already generated. I don’t want to waste a long generation, but I also don’t want a network hiccup to erase an answer that the user might still expect to find when they reconnect.
What should happen when the connection closes?
For a normal, one-off chat request, a client disconnect is usually a good reason to cancel the upstream generation. Detect that the response connection has actually closed, then pass a cancellation signal to the HTTP request or provider SDK. In a Node.js service using fetch, an AbortController can carry that signal. Some SDKs have their own cancellation options, so it’s worth checking the provider’s documentation rather than assuming that closing your app’s response cancels the model request.
Here’s the shape of that logic with a generic streaming HTTP endpoint. The URL, authentication, request body, and stream format will differ by provider; this example focuses on connecting the client connection to an upstream cancellation signal.
async function streamAnswer(req, res) {
const controller = new AbortController();
// Cancel upstream work only when the response connection closes early.
res.on("close", () => {
if (!res.writableEnded) {
controller.abort();
}
});
try {
const upstream = await fetch(process.env.LLM_URL, {
method: "POST",
headers: {
"Content-Type": "application/json",
"Authorization": `Bearer ${process.env.LLM_API_KEY}`
},
body: JSON.stringify({ prompt: req.body.prompt, stream: true }),
signal: controller.signal
});
if (!upstream.ok) {
res.writeHead(upstream.status);
res.end("The model request failed.");
return;
}
// Forward the provider's stream chunks while the client is connected.
res.writeHead(200, { "Content-Type": "text/event-stream" });
const reader = upstream.body.getReader();
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
if (!res.destroyed) res.write(Buffer.from(value));
}
} finally {
reader.releaseLock();
}
if (!res.destroyed) res.end();
} catch (error) {
// An intentional disconnect is not the same as a provider failure.
if (controller.signal.aborted) return;
if (!res.headersSent) {
res.writeHead(502);
res.end("Unable to get a response from the model.");
} else if (!res.destroyed) {
res.end();
}
}
}This is only a starting point: production code should also handle provider-specific stream events, timeouts, and errors. The key idea is to propagate request cancellation signals through every layer that supports them. Otherwise, your web server may stop sending bytes while the provider continues generating.
When cancellation is the wrong choice
There are cases where I might want generation to continue after the browser goes away. For example, a user may deliberately start a report, close the app, and return later. That’s better treated as a background task than as an abandoned chat stream: save a job ID, record its state, and let the client reconnect or retrieve the finished result. A background task also needs a clear limit on how long it can run and what happens if the user cancels it explicitly.
For an ordinary chat, I’d be cautious about silently turning every disconnected stream into a background generation job. It can surprise users and increase spend. If keeping work alive is a product feature, make that behavior visible and give users a way to stop it.
What about the partial answer?
There’s a useful distinction between sending chunks to the browser and saving them as a completed answer. If the connection drops halfway through, the partial text may be incomplete or misleading. You can discard it, or store it as an interrupted draft with a status that makes its state obvious. Either way, don’t mark it as a finished assistant response just because some tokens arrived.
If you do persist partial output, consider what the user sees on reconnect: can they resume the stream, read the partial draft, or start over? That choice affects your partial output persistence rules and how you represent message status in storage. It’s also helpful to keep the streaming response lifecycle separate from the message record, so a dropped connection doesn’t accidentally look like a successful completion.
- For ephemeral chat, cancel on disconnect and mark the response as interrupted.
- For an explicit long-running task, keep it running under a job ID and report its status.
- Track provider completion separately from whether the client received the full stream.
- Record usage and cancellation outcomes so you can compare expected and actual costs.
One detail I’m still checking is how much a cancellation saves. It should prevent further generation if the provider honors the abort, but it may not reverse usage already incurred or guarantee a particular billing adjustment. I’d log the request ID, when cancellation was requested, whether the upstream connection ended, and the provider’s reported usage where available. Those usage billing records can help distinguish a successful cancellation from a UI disconnect that left generation running.
How are others handling client disconnect cancellation in production—do you always stop the model, or do you preserve work for reconnecting clients? I’m especially interested in how different provider SDKs handle cancellation during a live stream, and what status you show for an answer that was only partly generated.
