# Benchmark

Source: https://getbusbar.com/docs/benchmark/
Last modified: 2026-08-05T17:57:34-07:00

Two questions get confused, so this page keeps them apart. **Added latency** is the end-to-end, wall-clock cost a calling app actually experiences (through-gateway minus direct-to-upstream); it’s measured identically for every gateway and reported on the [performance page](https://getbusbar.com/performance/) and the `/vs` comparisons. This page is about the *other* number: Busbar’s **per-request processing time**, the microseconds Busbar itself spends parsing your request, translating it, and serializing the response, *not* the network and *not* the model. It’s the CPU slice *of* the added latency, a compute/efficiency figure, not the latency itself, so we never rank gateways by it. Busbar measures exactly that and reports it in-band on every response (when enabled; see below), in a standard header:

```http
Server-Timing: busbar;dur=0.087
```

`busbar;dur` is total request time minus the upstream round-trip: everything Busbar did, nothing the upstream did. It’s a per-request number on Busbar’s own clock, so it is valid at the tail (each request is its own paired measurement) and independent of the upstream’s speed, which is what lets us measure the *same* number against an instant mock and against the real Anthropic API.

Every result below changes exactly one variable. If a benchmark moves two things at once, none of its numbers mean anything.

## Throughput: it scales with cores, linearly

Busbar’s request path is CPU-bound on the work that matters (parse, translate, serialize) and nothing else (no GC, no database in the hot path, no per-request lock) so throughput tracks the cores you give it, in a straight line. On one **`c7g.8xlarge`** (32 vCPU Graviton3, no hyperthreading, so one vCPU is one real core), driving **unique** requests so every one is real proxy work (not a cache hit), with the load generator and mock pinned to their own dedicated cores **on that same box** (8 cores each). Because the load harness shares the box with Busbar, this per-core rate is deliberately **conservative**: a single-box linearity demo, not Busbar’s isolated per-core ceiling (that clean ceiling comes from the onthebench field run, Busbar alone on a 4-core pin). The rows below are read live from the same `metrics/scaling.json` that drives the [performance page](https://getbusbar.com/performance/), the single source of truth for this number:

| Busbar cores                       | 2      | 4      | 6      | 8       | 10      | 12      | 14      | 16          |
| ---------------------------------- | ------ | ------ | ------ | ------- | ------- | ------- | ------- | ----------- |
| req/s (100% success)               | 21,626 | 47,880 | 74,514 | 103,984 | 132,313 | 161,630 | 187,510 | **209,214** |
| processing time (`busbar;dur` p99) | 35 µs  | 33 µs  | 33 µs  | 34 µs   | 32 µs   | 33 µs   | 32 µs   | 31 µs       |

**Linear at \~13,076 req/s per core; per-request processing time stays flat**, because it’s per-request CPU work, independent of load. This sweep is a **1.5.1** run on a `c7g.8xlarge` (Graviton3), measured 2026-08-03 (linear fit R²≈0.998); the per-request latency breakdown further down is a separate, earlier **v1.0.0** run on a smaller `c7g.2xlarge`, so read the two as different measurements of the same binary family rather than one continuous benchmark. The full harness and the identical Bifrost sweep are public and reproducible in [`GetBusbar/benchmarking`](https://github.com/GetBusbar/benchmarking). Below is how we isolate that per-request cost itself.

* [Result](#tab-panel-0)
* [Reproduce](#tab-panel-1)

## The setup (identical for every row)

**The units, up front**, so every number reads the same way:

* **`µs`**: microsecond. 1 µs = 1/1,000,000 of a second (and 1 ms = 1,000 µs). All latency is in µs.
* **`tok`**: token, the unit LLM APIs bill in (roughly 4 characters of text each). All payload sizes are in tok.
* **p50 / p99 / p99.9**: percentiles. p50 is the median (half are faster); p99 means 1 in 100 is slower; p99.9 is the tail, 1 in 1,000.

One AWS **`c7g.2xlarge`** (8 vCPU, Graviton3) in **us-east-1**, Amazon Linux 2023, running the released **`v1.0.0`** `aarch64` binary. Busbar is `taskset`-pinned to **2 dedicated cores**; the load generator and the mock get *different* cores. The headline metric is Busbar’s own `Server-Timing: busbar;dur=` value, captured per request; Busbar’s CPU is logged from `/proc`. Each cell is thousands of requests at concurrency 1 (the per-request floor).

## Result 1: processing time, swept by payload (Anthropic, vs mock)

Hold the protocol (**Anthropic**) and the upstream (**instant mock**) fixed; vary the **payload**. Two paths: a same-protocol passthrough, and a full cross-protocol translation (Anthropic in → OpenAI upstream → converted both ways). `busbar;dur`, in **µs**:

| Path                                            | Payload   |  p50 |   p99 | p99.9 |
| ----------------------------------------------- | --------- | ---: | ----: | ----: |
| **Same protocol** (Anthropic→Anthropic)         | \~8 tok   | `38` |  `45` |  `51` |
|                                                 | \~1k tok  | `41` |  `49` |  `57` |
|                                                 | \~12k tok | `93` | `137` | `147` |
| **Cross-protocol translate** (Anthropic↔OpenAI) | \~8 tok   | `48` |  `55` |  `62` |
|                                                 | \~1k tok  | `52` |  `60` |  `65` |
|                                                 | \~12k tok | `84` | `158` | `174` |

**It scales with bytes, not surprises.** A tiny call is `38`µs; a full cross-protocol translation of a \~12k-token (about 32 KB) body (parse, internal representation, re-serialize into a *different* protocol, both directions) is still `84`µs. A fixed per-request floor plus the bytes Busbar parses and serializes; no allocation cliff. Busbar’s CPU stays **\~`0.5` of one core** at concurrency 1.

**The tail stays tight.** Busbar is a single Rust binary with **no garbage collector**, so nothing in the request path pauses to sweep memory. Its p99.9 is only \~1.3 to 1.6× its p50 on small/medium payloads (and under \~2× even on the 12k-token translate); the processing time does not blow out into a tail. A garbage-collected proxy (Python/Node/JVM) pays an occasional GC pause on *some* requests, swelling p99/p99.9 by far more even when the median looks fine; we report p99.9 precisely so that’s visible.

**Throughput**, so “under one core” isn’t misread as a cap: saturating its 2 pinned cores Busbar sustained `19,505` req/s (\~`9,750` per core), and the Tokio runtime scales across cores.

## Result 2: processing time by protocol (all six, vs mock)

Now hold the **upstream** (mock) and **payload** (\~1k tok) fixed and vary only the **wire protocol**: same-protocol passthrough on each of Busbar’s six. This explains why it isn’t a single number: different formats cost different amounts to parse and serialize. `busbar;dur` p50 / p99, **µs**:

| Protocol  |  p50 |  p99 | note                    |
| --------- | ---: | ---: | ----------------------- |
| OpenAI    | `37` | `43` |                         |
| Anthropic | `42` | `49` |                         |
| Responses | `37` | `44` |                         |
| Cohere    | `37` | `44` |                         |
| Gemini    | `40` | `47` |                         |
| Bedrock   | `63` | `76` | + SigV4 request signing |

They cluster in the same tens-of-µs band; Bedrock is the outlier because it’s the one protocol that cryptographically signs every upstream request (AWS SigV4, an HMAC-SHA256 chain over the body), which is real CPU the others don’t pay. That’s the cost of speaking Bedrock natively.

## Result 3: invisible in production (paired, vs real Anthropic)

Change the upstream to the **real `api.anthropic.com`** and measure it: direct and through-Busbar requests are **interleaved** (direct, Busbar, direct, Busbar, …) in the same window, so Anthropic’s minute-to-minute drift hits both equally and **cancels** in the delta. Small \~8 tok calls to `claude-haiku-4-5`, **median client wall-clock**:

| Path (paired, interleaved)                       | p50 wall-clock |
| ------------------------------------------------ | -------------: |
| **Direct** (client → Anthropic)                  |        `807ms` |
| **Through Busbar** (client → Busbar → Anthropic) |        `790ms` |
| **Difference**                                   |    **`−17ms`** |

Through Busbar was 17ms *faster* at the median. That of course doesn’t mean Busbar speeds up Anthropic. It means Busbar’s cost is below the provider’s own run-to-run noise: in a paired test where drift cancels, the difference lands on *either* side of zero. Measure the two sides in separate passes and the provider’s own minute-to-minute drift masquerades as Busbar overhead; alternate them request-by-request and that drift hits both equally and subtracts out. We report only the median: at p99 the paired sample (n≈150 real calls) is dominated by provider-tail jitter, not Busbar.

And on those same requests, Busbar’s own self-reported number: `busbar;dur` p50 = `84`µs, higher than Result 1’s `38`µs for the same \~8 tok same-protocol shape because this run measures against the **real** `api.anthropic.com` (live TLS/connection handling on that hop) rather than Result 1’s loopback mock; `busbar;dur` still excludes the upstream round-trip itself, but the request/response handling around a real remote connection isn’t free. That’s still the actual compute, about 0.01% of a \~790ms model call, and you can read it on your own traffic.

Busbar is a proxy, so it adds one network hop (client to Busbar to provider). Here client and Busbar are co-located, so that hop is sub-millisecond and lost in the noise above; co-locate Busbar with your app in production and it stays sub-millisecond. The cost isn’t zero: it’s small, and measured both ways.

One self-cleaning script on one named instance type; the upstream is the only thing that changes between Result 1, Result 2, and Result 3.

**1.** Launch a `c7g.2xlarge` (Graviton3, arm64), Amazon Linux 2023, `us-east-1`.

**2.** Build the released binary, stand up a tiny instant mock that speaks all six protocols, pin the cores:

```bash
curl -fsSL https://getbusbar.com/install.sh | sh   # or build the tagged source


# mock: a small Go HTTPS server returning a valid response for each of the six wire formats,


#       sized to the request's max-tokens; served over a real Let's Encrypt cert on a loopback host


#       (127-0-0-1.local-ip.sh → 127.0.0.1) so the release binary's public-CA-only upstream TLS trusts it.


taskset -c 0,1 ./busbar ... &     # Busbar -> 2 dedicated cores


taskset -c 6,7  ./mock  ... &     # mock   -> its own cores
```

**3.** Fire requests pinned to the remaining cores and read the `Server-Timing: busbar;dur=` header off each response; that *is* the metric. For Result 3, **interleave** direct and through-Busbar requests so provider drift cancels. The full self-contained script (launch → pin → measure → log CPU → tear down) is published with the release.

Why these choices

* **`Server-Timing`, not a client stopwatch**: Busbar’s own clock excludes the upstream round-trip and the client’s jitter, and it’s a W3C-standard header you can read in DevTools or any APM tool, on your own production traffic.
* **A fixed cloud instance, pinned cores, logged CPU**: reproducible by anyone; the number is the cost of one path of work, not a big box. “I used my MacBook” is not a citable number.
* **Interleaved for the real-provider comparison**: the only way to separate Busbar’s cost from a provider whose own latency drifts minute to minute.

## The head-to-head method (comparing against another gateway)

The section above measures Busbar’s *own* per-request processing time. Comparing against another gateway is a different question, and it has its own rule: **change exactly one thing.** Same machine, same mock upstream, same load tool, same rates: the *only* variable is which gateway is in the path. If a comparison moves two things at once, its numbers mean nothing.

**We run the whole field, neutrally, in the open.** That method lives in [`GetBusbar/benchmarking`](https://github.com/GetBusbar/benchmarking), a public harness that runs **every** gateway (Busbar and a dozen others) the *same* way: one fresh, identical Graviton box per gateway, pinned to the same cores, fronting the same fast mock, gated on the same error/latency thresholds. It takes no side: gateways are ordered by measured result, colored by implementation language, and any gateway that can’t hold the endpoint is recorded, not hidden. The full standings (added latency, sustained throughput, and memory for all of them) are on the [performance page](https://getbusbar.com/performance/). Clone the harness and reproduce any row yourself.

**The mock is ours, fast, and the same for everyone.** We benchmark gateways, not mocks. A slow mock bounds every gateway to the mock’s speed, which hides the gateway’s real cost and flatters the slower one; a fast (\~0 ms) mock is never the bottleneck, so what’s left in the number is the gateway itself. We put the *same* fast mock in front of each gateway, and we don’t inherit anyone else’s mock as our ceiling.

**How we treat a competitor’s published numbers, believe the unflattering, verify the incredible:**

* If a competitor’s own published number is *unflattering*, we take it at their word. There’s nothing to learn by spending a machine to prove a competitor is slow. That’s why the [LiteLLM comparison](https://getbusbar.com/vs/litellm/) uses their published overhead and throughput as-is, clearly labelled as theirs.
* If a published number is *too good to be true*, we don’t accept it: we reproduce it. Bifrost publishes an “11 µs overhead” that their own benchmark tool never emits and their own `/metrics` can’t resolve (histogram buckets start at 5 ms). So for the [Bifrost comparison](https://getbusbar.com/vs/bifrost/) we ran **their own benchmark** (their tool, their config, their mocker, co-located exactly as their docs prescribe) and report what it actually produces. If Bifrost tells us how the 11 µs is instrumented, we’ll run it on the same box and publish what we get.

**Reproduce the Bifrost head-to-head yourself**: their tool, their documented setup:

```bash
# One t3.xlarge (4 vCPU), CpuCredits unlimited. Co-located: mock + gateway + load tool on one box.


git clone https://github.com/maximhq/bifrost-benchmarking && cd bifrost-benchmarking


( cd mocker && go run main.go -port 8000 )        # their 0 ms mocker


go build benchmark.go                              # their load tool






# Gateway A: Bifrost, their documented t3.xlarge config (initial_pool_size 15000 / buffer_size 20000),


#             OpenAI provider pointed at the mocker (network_config.base_url = http://127.0.0.1:8000):


docker run --network host --pid host maximhq/bifrost:v1.6.4


./benchmark -provider bifrost -rate 1000 -duration 300 -output bifrost_1000.json


./benchmark -provider bifrost -rate 3000 -duration 180 -output bifrost_3000.json






# Gateway B: swap ONLY the gateway. Busbar release binary, openai provider → http://127.0.0.1:8000,


#             governance + logging off. Same tool, same rates, same box, same mock:


busbar &   # BUSBAR_CONFIG pointing at the mock; then re-run the same ./benchmark lines.
```

Subtract nothing: the tool reports end-to-end latency, memory, and success rate per run. The raw results for our runs are public in [`GetBusbar/benchmarking`](https://github.com/GetBusbar/benchmarking). Every number on the comparison pages reads from one version-stamped source (`facts.ts`); re-run on a new release, update `facts.ts`, and the pages re-derive.

**Why a standard header.** `Server-Timing` is W3C-standard, so you can read `busbar;dur` in browser DevTools or any APM tool, on your own production traffic, with no special tooling. The overhead number isn’t a one-time lab claim; it’s something you can verify continuously, on your own requests.

**Enabling it.** The `Server-Timing` header is opt-in (default off) to avoid fingerprinting Busbar on unauthenticated responses. Set `advanced.response_headers.server_timing: true` in `config.yaml` to turn it on.