sdlc.dev

AI engineering path · Models in your system · Topic 7 of 12

Streaming responses to users

Not started

Without streaming, your code waits for the model to finish the whole answer and then receives it in one response. With streaming, the provider sends the answer as it is generated, as a series of small events, most commonly over server-sent events (SSE). The total time is about the same. What changes is time to first token: users see words appear within a moment instead of staring at a spinner, which matters a lot for chat and long-form answers.

Why it matters: A long answer can take many seconds to generate in full. Streaming shows the first words almost immediately, which changes how fast the feature feels, but it also changes how you handle errors, validation and cancellation.

  1. 1Time to first token

    Streaming does not make generation faster; it lets users start reading while the rest is produced.

  2. 2Events, not one body

    The provider sends a sequence of small events (text deltas, tool-call fragments, usage, a final stop reason) over server-sent events or a similar protocol.

  3. 3Errors arrive mid-stream

    A stream can fail after half the answer is shown. Plan what the user sees when that happens.

  4. 4Cancel what nobody reads

    When the user navigates away or presses stop, abort the upstream request so you stop paying for tokens.

1

How it works

The explanation, with examples you can run.

What arrives

A stream is a sequence of typed events. Names differ by provider, but the shape is similar:

  • a start event with ids and metadata,
  • many small deltas: pieces of text, and for tool calls, fragments of the arguments' JSON,
  • a final event with the stop reason and token usage,
  • occasionally an error event, and sometimes keep-alive pings.

Official SDKs assemble these for you and can hand you either each delta or the finished message. If you parse events by hand, accumulate tool-call argument fragments and only parse them once that block is complete.

The fake stream below emits deltas on a timer, the way a real one does, and shows the two numbers that matter for perceived speed.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Relaying a stream through your backend

Keep the provider key on your server. The browser calls your endpoint; your server authenticates the user, checks limits, calls the model with streaming on, and forwards text to the browser as it arrives. Propagate cancellation in both directions: if the browser disconnects, abort the upstream request.

// Placeholder client; a server route that relays model text as server-sent events.export async function POST(req: Request) {  const user = await requireUser(req)  const { messages } = await req.json()  const upstream = await llm.stream({ model: MODEL, messages, signal: req.signal })   const body = new ReadableStream({    async start(controller) {      const send = (data: unknown) => controller.enqueue(new TextEncoder().encode(`data: ${JSON.stringify(data)}\n\n`))      try {        for await (const ev of upstream) {          if (ev.type === "text_delta") send({ text: ev.text })        }        send({ done: true })      } catch (err) {        send({ error: "interrupted" }) // the UI marks the answer as incomplete        log.error({ user: user.id, err })      } finally {        controller.close()      }    },  })  return new Response(body, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" } })}

Things streaming makes harder

  • Validation. You cannot validate an answer you have not finished receiving. Stream to the screen if you like, but run output checks (schemas, content filters, link checks) before saving, acting or sending anything further.
  • Rendering. Markdown arrives half-formed (an open code fence, half a link). Use a renderer that tolerates partial input, and sanitise as you would any untrusted HTML.
  • Errors after success. HTTP status is sent before the body, so a 200 can still end in an error event or a dropped connection. Show the answer as interrupted and offer a retry.
  • Infrastructure. Proxies and serverless platforms may buffer responses or close idle connections; check timeouts, disable buffering for the route, and send occasional keep-alives on long answers.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Your feature returns structured JSON that the UI renders as a form. Should you stream it?
  2. 2.The stream breaks after 300 of an expected 800 tokens. What is a reasonable behaviour?
  3. 3.Where should the provider's API key live in a streaming chat feature for a web app?

0 of 3 right so far. Get all three to finish this topic.

Go deeper