Streaming

Keiro streams completion responses as server-sent events. Set stream to true, process events in order, and keep the connection open until the surface's terminal event or in-band error.

On this page 1 of 11

Use curl -N when testing from a terminal so client-side buffering does not hide incremental delivery.

During a long stream, the gateway writes a frame at least every 10 seconds. On Chat Completions and Responses that is the static SSE comment : keepalive, about every 8 seconds, and during a silence an idle chunk or event right after it (see Silence while the model thinks). On Messages it is the native ping event, and during a silence inside an open content block an empty delta in that block right after it. SSE comments carry no model output, consume no event sequence number, and are not replayed; conforming parsers ignore them while the bytes keep transport-idle timers alive. Events carry no SSE id: field; a Responses event's position is its sequence_number.

Chat Completions stream#

Chat Completions sends data: chunks with text in choices[0].delta.content. Tool-call fragments appear in choices[0].delta.tool_calls. Assemble by choice and tool-call index.

A tool call streams as it is generated: the first tool_calls chunk carries the call's id and function.name as soon as the model commits to the call, and later chunks carry function.arguments fragments; concatenate the fragments per index. Fragment boundaries are not stable. If the output limit ends the response while a call is still streaming, the fragments already delivered stand, no further tool_calls chunk follows, and the terminal choice carries finish_reason: "length"; such a call is not complete and must not be executed.

While the model works without sending anything, a chunk carries one space in choices[0].delta.reasoning_content, so a client's stall timer sees progress. It comes at any point in the stream: before the first text or tool call, after text, between a tool call's argument fragments, and while the stream is finishing. The first comes within about 12 seconds of the last output, then about every 8 seconds. Keiro writes this value, not the model, and it is not a readable status. It is not billed and is never part of delta.content. Some clients show it as an empty reasoning block, which you can hide or ignore. If your client saves the field and sends it back as reasoning_content on the assistant message, Keiro drops it.

A timer that trims whitespace, or that counts only content or tool_calls, does not count this value.

The successful stream ends with a terminal choice and data: [DONE]. With stream_options.include_usage: true, the chunk with the terminal choice carries no usage, and one more chunk with choices: [] and the request's final usage follows it, before data: [DONE].

Runnable streaming samples live on Chat Completions.

Responses stream#

Responses emits typed events. Each event is sent under an SSE event: line equal to its type. Read type and ignore event types your client does not handle.

Event typeMeaning
response.createdThe response exists and streaming started
response.in_progressGeneration is active
response.output_item.addedA typed output item started
response.content_part.addedA content part started
response.output_text.deltaIncremental text in delta
response.output_text.doneOne text segment finished
response.function_call_arguments.deltaFunction arguments arrived for a tool call
response.function_call_arguments.doneA tool call's arguments finished
response.output_item.doneA typed output item finished
response.completedSuccessful terminal event
response.incompleteTerminal event for an early stop, such as an output limit
response.errorIn-band stream failure
response.failedStandard failure terminal, right after response.error once response.created was sent
keepaliveThe stream is alive during a silence; it carries no output

Successful Responses streams also finish with data: [DONE] after the terminal event.

A function_call item is announced with response.output_item.added (status: "in_progress", empty arguments) as soon as the model commits to the call, and its arguments arrive through response.function_call_arguments.delta frames while the model is still generating; accumulate by output_index. A complete call closes with response.function_call_arguments.done and response.output_item.done (status: "completed"); the concatenated deltas equal the arguments of both close frames and of the item in the terminal output, byte for byte, and the item keeps the id and call_id it was announced with. If the output limit ends the response mid-call, the item closes with response.output_item.done carrying status: "incomplete" and the arguments text delivered so far, the terminal event is response.incomplete with incomplete_details.reason: "max_output_tokens", and the same item appears in the terminal output; an incomplete call must not be executed.

Runnable streaming samples live on Responses.

Messages stream#

Messages emits message_start, content-block start/delta/stop events, message_delta, and message_stop. The terminal message_delta carries the final stop reason and usage. Tool calls use tool_use content blocks and input JSON deltas. When thinking summaries are returned (enabled thinking unless display is omitted, or adaptive thinking with display set to summarized), unsigned thinking blocks stream first, through thinking_delta events, and close before the text block opens.

A tool_use block opens with content_block_start (its id and name, empty input) as soon as the model commits to the call, and its input arrives through input_json_delta events while the model is still generating; a text block that was open closes first. If max_tokens ends the response mid-call, the block closes with content_block_stop holding only the input fragments delivered so far and the terminal message_delta carries stop_reason: "max_tokens"; such a block is not a complete call and must not be executed.

See Messages for supported request fields and content shapes.

Stream lifetime#

A request lives as long as bytes keep flowing. Keiro never ends a stream, or a non-streaming request, for taking long: a model that keeps working, including long silent reasoning that sends progress or keepalive events upstream, runs until it finishes. Total work is bounded by the output-token budget, not by a clock.

A run ends early in three cases only:

  • The upstream stops making progress. If the model's provider sends nothing at all for 10 minutes, the run is cancelled with the retryable gateway_timeout error: an error frame on a stream, or the error body on a non-streaming request. Keiro may retry a stalled model call before it returns the error, so the error can arrive well after the first 10 minutes of silence. Tokens consumed before the stop are billed.
  • The client leaves. When your connection closes, Keiro stops the in-flight model calls on most model paths. On the rest, a call can keep running for a while, and its result is discarded. A stream that returned no output before it closed is not billed. Otherwise, tokens already consumed are billed.
  • The service restarts. A restart or deploy can end a run that is still going, with the retryable gateway_restarting error: an error frame on a stream, or the error body on a non-streaming request that has already sent its status.

Long requests#

Keiro keeps every open connection alive with bytes: an SSE comment at least every 10 seconds on a stream, and a space every 15 seconds once a non-streaming reply has sent its status. Set one read (idle) timeout of 3,720 seconds (3,720,000 ms) and no total timeout.

Some clients, such as Codex and Claude Code, count only SSE events and ignore comments. With this setting they wait through up to 62 minutes of silent reasoning.

Silence while the model thinks#

A model can think for minutes before its first output, and again after output has started. What your client receives in that time depends on the surface:

SurfaceBefore output startsAfter output has started
Chat Completions: keepalive comments, and a chunk with one space in delta.reasoning_content: keepalive comments, and the same chunk with one space
Responses: keepalive comments, and response.in_progress again: keepalive comments, and a keepalive event
Messagesping events, and on a request that enabled thinking an empty thinking blockping events, and an empty delta in the open content block

Output starts with the first text or tool call on Chat Completions, with the first output item on Responses, and with the first text or tool_use block on Messages. Some frame arrives at least every 10 seconds, and on Chat Completions and Responses a : keepalive comment arrives about every 8 seconds, as before. The Chat Completions chunk and the Responses event arrive right after a comment, the first within about 12 seconds of the last output and then about every 8 seconds.

On Responses, response.in_progress is repeated only before the first output item opens; after that the event is keepalive. While a custom tool call has been announced and its input has not started, the event is instead an empty response.custom_tool_call_input.delta on that call. The short wait while a finished response's final event is held for the stream to close carries comments only.

Where Keiro has turned on the idle reasoning item, a Responses stream that is silent before its first output item instead opens one reasoning item of Keiro's own at output index 0, with an id that starts rs_idle_, and each idle event is a response.reasoning_summary_text.delta with one space. The item closes before the model's first item, which then starts at index 1, so find items by type, not by position. Its summary is whitespace only. It is not the model's reasoning: Keiro recognises it by its id and drops it when you send it back in input (whole, or as an item_reference to that id) or through previous_response_id, and it adds nothing to usage. A non-streaming reply never contains it.

On Messages, while a text, tool_use or thinking block is open, each ping during a silence is followed by an empty delta of that block's own kind on its index: text_delta with text: "", input_json_delta with partial_json: "", or thinking_delta with thinking: "", the first within about 12 seconds of the last event and then about every 8 seconds. It adds nothing to the block's text, input or thinking, and leaves a thinking block's signature unchanged. Between blocks, and while a finished stream is closing, Messages sends ping events only.

Before the first block opens, a Messages request whose thinking.type is enabled or adaptive gets one empty thinking block, at index 0. It opens with content_block_start carrying {"type": "thinking", "thinking": "", "signature": ""} right after the ping of the first tick that finds the stream silent, about 8 seconds after message_start, so any wait of more than about 8 seconds carries it, and the answer's index depends on latency. Each tick after it brings an empty thinking_delta right after its ping. It has no text, and its signature is "". If the model's own thinking follows, it continues in that block. Otherwise the block closes with content_block_stop right before the first text or tool_use block, which then starts at index 1. A non-streaming reply does not carry it. It may be sent back as history, with signature: "" or without the key; it carries nothing to the model. Claude Code's stream watchdog counts a ping only as a heartbeat, at most 30 in a row, so this block is what keeps a long think from being abandoned. A request without thinking gets ping events only before its first block.

None of these carry output: ignore them. An SSE parser reports the chunks and the events, and skips the comments. A client that turns on the OpenAI SDK's strict response validation, which is off by default, rejects the keepalive event, because the SDK's event types do not include it.

If your client's idle timer is shorter than the model's thinking, the client closes the connection first. The work is lost: Keiro stops the model call on most model paths and discards any result. A retry starts over and can hit the same timer. Repeated failures at about the same age, with no error from Keiro, usually mean this timer. Check the timers in your HTTP client, SDK, agent harness and any proxy in the path.

Set the idle timer to 3,720 seconds as above. If you cannot, ask for less thinking with a reasoning effort: reasoning_effort on Chat Completions, reasoning.effort on Responses, or output_config.effort on Messages. It is a ceiling on the reasoning eb1 orchestration chooses, so a lower value shortens the silence but does not fix its length.

On a stream, retry automatically only when no response byte has arrived, and send the same Idempotency-Key with the retry. Once any stream byte has arrived, do not resend the request automatically. A non-streaming body that ends before its JSON is complete delivered no answer; see the next section.

Long non-streaming requests#

A non-streaming request whose reply is not ready after 30 seconds, and that has been sent to a model, receives its 200 status and headers, then one space character every 15 seconds, then the JSON body. The status comes at the later of those two moments. A refusal decided before the request reaches a model keeps its own status, however long the checks before it take.

Leading whitespace is valid JSON, so standard JSON parsers read the body unchanged, and every byte resets your client's read timeout: the OpenAI and Anthropic SDK defaults keep reading for as long as the model works. The early status carries the same request id, rate-limit and CORS headers as a reply that finishes quickly, so browser clients read it the same way.

Because the status is sent before the body, a failure after the status was sent cannot change it. What arrives depends on whether a retry may fix the failure:

  • A failure a retry may fix, such as an upstream fault, a stalled model call or a gateway restart, closes the connection before the body is complete. Your client raises a connection error, and the OpenAI and Anthropic Python SDKs retry it with their usual backoff; each retry runs the request again from the start. The OpenAI and Anthropic Node SDKs raise the error without retrying. The failure's message and code do not reach your client: the X-Request-Id header sent with the 200 identifies the request for support.
  • A failure a retry cannot fix arrives as the same error JSON body the endpoint returns for that failure, under the 200. Check for a top-level error object (Messages: "type": "error") before reading a completion.

A request that finishes or fails before its status is sent is answered with its own status, as a fast reply is. A stream reports a late failure in the stream itself, as the next section describes.

Handle in-band errors#

Once response headers and the first body byte are sent, the HTTP status cannot change. A later failure therefore arrives in the stream.

  • Chat Completions places type, code, message, and optional retry_after_seconds on the error frame. A failure an upstream reports mid-stream has code provider_error, and its type names the class: rate_limit_error, invalid_request_error or api_error.
  • Responses and Messages place those fields under error.
  • On Responses, once response.created was sent, a response.failed event follows response.error. Its response repeats the id, model and created_at of response.created with status: "failed", and its response.error is the same object as the error frame's error. Clients that read only the standard terminal events, such as Codex, see the failure there. A stream you cancel ends with its cancelled frame only.
  • response.failed carries in response.output every item whose response.output_item.done was sent, as that event carried it, in output_index order. An item that was still open when the stream failed is left out, so output is empty when no item had finished, and an item's position in output can differ from its output_index.
  • On Messages, error.type is the Anthropic type for the failure's status, the same type the non-streaming error body carries: overloaded_error for 503, timeout_error for 504, rate_limit_error for 429, invalid_request_error for 400, and api_error for other server failures. code names the failure precisely.

Treat an error frame as terminal. Do not wait for normal completion after it. See Errors for the shared code taxonomy.

Verify delivery#

Run a curl request Shell
printf 'Keiro API key: '
IFS= read -rs KEIRO_BEARER
printf '\n'

curl -sS -N https://api.keirolabs.ai/v1/responses \
  -w '\nTTFB=%{time_starttransfer}s total=%{time_total}s http=%{http_code}\n' \
  -H "Content-Type: application/json" \
  -d '{
    "model": "eb1-preview",
    "stream": true,
    "input": "Count from one to five, one number per line."
  }' \
  -H @- <<<"Authorization: Bearer $KEIRO_BEARER"

Time to first byte should normally be lower than total time. Similar values can mean the response was short or that a client, proxy, or network path buffered the stream.

Retry safely#

Requests retain at most 600 images counted across every placement (pasted images and tool-result screenshots together), with 10 MB (10,000,000 bytes) of encoded data per image, 24 MiB of encoded inline media per request (images and PDFs together), and 8 KiB per remote image URL. A request with more than 100 images can be refused before processing with the same measured error. PDF inputs allow 2 files, 8 MiB of encoded data each and 12 MiB total; filenames allow 255 characters. Replay items allow 512 KiB each (raw UTF-8 bytes for redacted thinking, serialized bytes for other replay items) and 4 MiB serialized bytes total. The content scan allows 65,536 nodes and a nesting depth of 64; it does not cap text length. These checks apply before both streaming and non-streaming responses. Media-limit errors use phrases such as “images exceed the API limit”; replay-size and content-node errors use “prompt is too long” with measured bytes or content parts, so clients such as Claude Code can automatically remove media or compact history. Depth-only failures remain count-free; these recovery phrases do not promise that an unchanged retry will succeed.

An interrupted stream can contain output that the user already saw. Retry only when repeating the logical operation is safe. Use an idempotency key for retry-sensitive requests, honor Retry-After, and avoid concatenating a retry onto partial output as if it were one response.

Search Keiro docs

Start typing to search pages and sections.

Start typing to search pages and sections.

Documentation

Console