Honest numbers

I build Busbar: the reliability and fidelity layer for multi-model AI traffic. It’s a single static Rust binary that sits between your application and your providers, routing across pools, translating losslessly between six wire protocols, and staying up through provider failures.

A control plane lives in your request path. So the first question anyone should ask is: what does it cost to pass through? I want to answer that the same way every time, with numbers you can reproduce, not adjectives. This post is about how we measure, and the one rule we hold to.

The rule: no number we can’t reproduce

Every performance fact on this site carries three things: what version it was measured on, what hardware, and how. If a number can’t be stamped with all three, it doesn’t ship. That’s not a slogan. It’s enforced. The site’s facts live in one file with a self-check that fails the build if any measurement is missing its source, version, or hardware. A page cannot quote a latency figure that isn’t traceable to a run.

There is exactly one canonical value for each thing we measure. The added-latency number the homepage shows is the same object the performance page shows, which is the same object the docs show. They can’t drift, because they’re the same variable. When we re-benchmark, the number changes in one place and everywhere at once.

What we measure, and the honest version of each

Added latency, the cost of being in the path. This is the CPU time Busbar itself spends: parse the request, translate the protocol, serialize the response. Not the network, not the model. We expose it per-request as a Server-Timing: busbar;dur span, so you can read it on your own traffic, and we measure it the same way. On a 16-core Graviton3 pin, a same-protocol small request costs p50 26 µs, p99 33 µs. The honest caveat is right next to it: a cross-protocol translation of a ~12k-token body is our worst published case at p50 84 µs, p99 158 µs. Bigger bodies cost more to re-serialize, and we publish the expensive case, not just the flattering one.

In-process handling latency. Separately, the busbar crate has a release-build test that runs 50,000 requests against an instant mock and records the handling distribution: p50 35.6 µs, p90 40.1 µs, p99 52.8 µs. It’s a regression metric: if a change makes the hot path slower, that test moves, and we see it before you do.

Throughput, and how it scales. On the same Graviton3 box, swept from 2 to 16 pinned cores with unique request bodies (so every request is real proxy work, not a cache hit), Busbar goes from ~19,800 req/s on 2 cores to ~156,000 req/s on 16, close to linear, roughly 9,750 req/s per core. And the added-latency stays flat across the whole sweep: p99 sits at 32–37 µs whether you’re on 2 cores or 16. Scaling out doesn’t cost you tail latency.

Memory. Idle resident set is about 8 MB. Under that 16-core throughput sweep the working set sits between ~58 and ~118 MB. Under a deliberately hostile soak (150 KB payloads, sustained) it plateaus at a bounded working set and falls back to ~250 MB within about 30 seconds of the load ending, instead of holding a high-water mark. Bounded, and it gives memory back.

Footprint. The Docker image is a FROM scratch container: the static binary plus the provider catalog, about 6 MB compressed. The raw binary is about 13 MB. No runtime, no interpreter, no sidecar in the request path.

Why “measured on the shipped binary” matters

It’s easy to publish a benchmark of a build that isn’t what you ship. So the numbers above come from the release artifact, and the three that can be are auto-measured from the actual Docker image and binary on every refresh (image size, binary size, idle memory) rather than typed in by hand. The latency and scaling runs are stamped with the exact hardware (c7g.8xlarge, 32 vCPU Graviton3, 16-core pin, us-east-1) and the version they ran on. When you read a number, you can see what produced it.

That’s also why you won’t find a single round, decorative figure here. “Microsecond overhead” is a category, not a measurement. The measurement is 26 microseconds at p50, 33 at p99, on a named box, at a stated concurrency, and 158 at p99 when the body is big and the protocols differ. Both are true. We publish both.

Read it on your own traffic

None of this asks you to trust the post. Turn on the Server-Timing header and you get Busbar’s own added latency on every response, in your environment, on your hardware, under your load. The /metrics and /stats endpoints expose the rest. The benchmark method is in the repo. If our numbers don’t reproduce for you, that’s a bug we want to hear about, the same way we’d want to hear about a wrong status code.

Performance claims age badly when they’re marketing. They age well when they’re instrumentation you can re-run. We chose the second kind on purpose, and we intend to keep every number on this site honest enough that you never have to take our word for it.

• Matthew