Benchmarks

Gateway overhead

What the gateway itself adds per request — routing, key auth, worker proxying, and metering — measured with DynamoDB and Cloud Map replaced by in-process fakes and a local stub worker, so AWS and network round-trips are excluded on purpose. Deployed latency adds your provider RTTs on top (in a live wiring test through Fly-shaped infrastructure, DynamoDB metering from the gateway region added ~250 ms end-to-end including a real Cloud Map lookup and cross-region DynamoDB transaction).

Reproduce with:

python scripts/bench.py

Measured 2026-08-04 on an Apple-silicon laptop, Python 3.14, single gateway process:

Scenario

Result

Added latency, 1 KB body (p50 / p95 / p99)

0.43 / 0.62 / 0.79 ms

Throughput, 1 KB body, 8 connections

~2 300 req/s

Throughput, 1 MB body, 8 connections

~765 req/s (~765 MB/s)

The numbers to take away: sub-millisecond median overhead, and body proxying runs at loopback memory speed — the gateway will not be the bottleneck in front of GPU scoring workers, whose forward passes are measured in tens of milliseconds.

Caching effects

Two small caches keep steady-state AWS traffic near zero on the hot path:

  • API-key auth: one DynamoDB GetItem per key per AUTH_CACHE_SECONDS (default 30 s) instead of one per request.

  • Worker discovery: one Cloud Map DiscoverInstances per DISCOVERY_CACHE_SECONDS (default 5 s) shared across all requests.

The metering transaction is the one intentional per-request AWS write — it is the billing record, so it is never cached or batched.