In-flight failover

When a lane fails, Busbar reroutes the request to another pool member before your client sees a byte, even mid-stream, across protocol families. This page covers the first-byte boundary, the per-request failover budget, context-length failover, session affinity, and what happens when a pool is exhausted.

Cross-references: Circuit breaker (how lanes trip) · Pools (structure) · Configuration (field reference).

The first-byte boundary

Failover is bounded by when the upstream starts streaming a response body to the client. Before the first upstream byte reaches the client, any transport or pre-response failure (connect error, timeout waiting for headers, transient upstream response) transparently fails over to another pool member. From the client’s perspective, the request is still in flight.

This pre-first-byte window covers the bulk of real provider failures: connect errors and timeouts, 429 rate-limit responses, and 5xx errors returned on the response headers all arrive before any body byte, so they fail over transparently. A failure only becomes unrecoverable once the upstream has already streamed a byte to the client and then dies mid-generation.

Why mid-stream failover is impossible: for every gateway, not just Busbar. A streaming response is a stateful continuation. Once a byte has been sent, you cannot un-send it: the client has already rendered those tokens. A replacement provider cannot resume the first provider’s half-finished generation either, it would start a brand-new completion from the prompt, so splicing its fresh output onto the partial stream produces duplicated or contradictory text. The only alternatives are to resend the whole response (the client sees tokens twice) or abandon the partial, neither is transparent. This is a property of streaming itself, so no transparent gateway (LiteLLM and OpenRouter included) does mid-stream failover; it is physics, not a missing feature.

The one real lever: a configurable pre-release buffer (on the roadmap, not yet built). The idea: hold the first K tokens / T ms of the upstream stream before releasing any byte to the client; if the provider dies inside that window, nothing has been sent yet, so Busbar can still reroute. The trade-off is up to T ms of added TTFT, so it would be opt-in per pool and default to off. It widens the failover window; it does not claim the impossible mid-stream splice above. Today Busbar has pure pre-first-byte behavior. See the roadmap.

After the first byte: failover is impossible (per the reasoning above). The client already holds a partial response body. If the upstream then fails mid-stream:

• For SSE responses (OpenAI, Anthropic, Gemini, Cohere, Responses ingress): Busbar emits an SSE error event to the client and closes the connection. The lane records the failure, which may trip its breaker.
• For non-SSE responses: the body stream terminates.

In both cases the client must detect the incomplete response and retry. The breaker will have recorded the failure, so a subsequent retry to the same pool is likely to be routed to a different member.

The practical implication: for workloads where mid-stream failure recovery matters, keep responses short or use non-streaming calls where the full response is buffered before delivery. For long streaming responses, implement client-side retry with session affinity disabled on retry (or send the retry to a different pool).

Failover budget and exclusions

Each request carries a per-request failover budget: a wall-clock deadline and a hop count cap. Both are configured per pool:

pools:
  resilient:
    members:
      - target: primary-model
        weight: 3
      - target: fallback-model
        weight: 1
      - target: last-resort-model
        weight: 1
    failover:
      timeout_secs: 30     # wall-clock budget across all hops; default 120
      max_hops: 3          # max hop count; default 3
      exclusions:
        - last-resort-model    # never selected as primary or failover

exclusions is a per-pool member blocklist. A model listed in exclusions is never selected through the pool: not as the initial pick and not as a failover destination. A request to the pool can never land on it. The model itself stays fully routable by its own name, because direct routing bypasses pools entirely: "model": "last-resort-model" on /v1/chat/completions, or POST /last-resort-model/v1/messages. That is the point of excluding rather than removing: the member keeps its /stats row (an expensive last resort a human or a specific job can still invoke deliberately), while the pool’s automatic selection never spends on it. Each exclusions entry must name a member of this pool.

Already-tried lanes are accumulated in an excluded set across hops for the lifetime of the request. A lane that succeeded (2xx headers) but whose body then failed before the first byte is refunded its max_requests budget spend and is also excluded from further hops on that request.

Catching hangs

Some providers fail by hanging: they accept the connection and never return response headers. The per-request failover budget does not catch this well, because the hang quietly eats the whole budget on one member before any hop can happen. Busbar closes that gap with a per-attempt cap on time to response headers, configured as attempt_timeout_ms. When the cap expires, the attempt is recorded as a transient breaker failure and the request hops to the next member immediately. Set it on a model as that model’s default, and override it per pool member: the same model can carry a 10s cap in a batch pool and a 50ms cap in a latency-critical one. The cap never cuts a stream that has started answering (it covers connect + headers only), and it is always floored by the request’s remaining failover.timeout_secs. Full semantics and examples in the configuration reference.

Context-length failover

When a request is too large for a member (the provider returns a context-length error), Busbar does not penalize the lane: it was healthy; the request simply did not fit. Instead, it excludes from this request’s candidate set any member whose declared context_max is ≤ the failed lane’s, then retries to a larger (or unknown-context) member.

pools:
  long-context:
    members:
      - target: claude-haiku
        context_max: 200000
      - target: gemini-2.5-flash
        context_max: 1048576

A member with no context_max set is never excluded on context-length grounds, it is always a candidate, and if it also rejects the request as too long, that rejection is still treated as a context-length failure (no breaker penalty) and the lane is simply excluded for the rest of this request.

Context-length failover is suppressed on 5xx responses, even if the body mentions a context-length-related code, to prevent a broken backend from dodging normal breaker penalties.

Session affinity

Pin a session to one member while it remains healthy:

pools:
  smart:
    members:
      - target: claude-sonnet
      - target: gpt-4o
    affinity:
      mode: session
      header_name: x-session-id    # default

When a request carries x-session-id: <value>, Busbar pins that session to a specific member. If the pinned member is unavailable (tripped, at-capacity, or excluded), affinity is ignored and normal SWRR selection runs, affinity is a preference, and an unhealthy member releases it. The client receives no signal that the pin was broken.

session is the only supported mode. header_name defaults to x-session-id.

Pool exhaustion

When all candidates are unavailable, tripped, excluded, or at-capacity, the pool is exhausted. The on_exhausted action decides what happens:

pools:
  primary:
    members:
      - target: fast-model
      - target: fallback-model
    on_exhausted:
      action: fallback_pool:overflow    # try another pool

  overflow:
    members:
      - target: cheap-model
    on_exhausted:
      action: least_bad    # degraded but not a hard error

Action	Behavior
reject / status_503 / 503	Return 503 with Retry-After set to the soonest member’s cooldown expiry. (Default when on_exhausted is omitted.)
least_bad	Select the member whose cooldown expires soonest and send the request anyway, even though its breaker is Open. Logs a loud degraded-service warning.
fallback_pool:<name>	Route to another named pool. Loop-guarded: if the fallback pool itself is exhausted and also falls back, cycles are detected and broken.

(The parser also accepts the spellings status503 for reject and least-bad / leastbad for least-bad.)

A 503 from pool exhaustion sets Retry-After so clients and upstream proxies know how long to back off. The /metrics counter busbar_requests_total{outcome="exhausted"} tracks these. A rising exhausted rate combined with a falling busbar_upstream_attempts_total for the pool’s lanes indicates breakers are tripping faster than they recover, check busbar_breaker_trips_total and /stats for individual lane state.

Multi-hop fallback chains, primary → overflow → emergency, work as long as they form a DAG (no cycles back to a visited pool). A self-referential or cyclic chain is rejected at config validation; a runtime loop is caught by the loop guard and results in a 503.

---