New benchmark: AI Gateway Model Index

ComputeSDK's new AI Gateway Model Index benchmark catalogues which models each gateway carries, how they're routed, and what they cost.

David Tice

The problem

We've been testing AI Gateway latency across the top 4 model families for almost two months now. We plan to release many more benchmarks for AI Gateways, but before we do that, we're releasing a much simpler (on the surface) benchmark: an index of models/prices across all AI Gateways.

We’ve built the source of truth for model availability, naming, pricing, and routing.

What we built

The AI Gateway Model Index benchmark is a daily catalog of every model exposed by each gateway's public model-list endpoint. It normalizes the responses into one machine-readable matrix and enriches pricing from each gateway's own pricing catalog where one exists.

It covers OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, BlazeRail, LLM Gateway, Pydantic AI Gateway, Concentrate AI, Ramp Router, Neon AI Gateway, ngrok AI Gateway, LLM API, and Github Copilot.

What it measures

For each model, the index captures:

  • Model id and display name
  • Owned-by / upstream provider
  • Supported routing options and provider pinning
  • Context length and max output tokens
  • Pricing, including per-provider pricing where exposed
  • Whether the gateway exposes a programmable model list at all

Failures are recorded, not hidden.

What it doesn't measure

We know that this is not a true benchmark for pricing. This benchmark does not take into account caching, dynamic routing, or any other methods a gateway can use to reduce costs, that benchmark is in the works. This benchmark is a first step in that direction.

Why it matters to you

  • Model availability. The same model may be on one gateway and missing from another, or exposed under a different name.
  • Compare real pricing. The index surfaces per-token and per-provider costs so you can find the cheapest way to call the same model.
  • Find model discounts. You may find that your favorite model is being offered for free from another router.
  • Keep routing control. Some gateways let you pin an upstream provider; others route dynamically. That affects latency, cost, and compliance.
  • Protect against catalog drift. Models are added, retired, and repriced daily. A fixed integration assumption goes stale quickly.
  • Build a fallback plan. Identify which other gateways can serve the same model if your primary changes terms or has an outage.
  • Shortlist before benchmarking. Use the model index to find gateways that carry your model, then use the latency benchmark to pick the fastest one.

Where to see it

The benchmark will be available in the ComputeSDK benchmarks once it is released publicly. Until then, it is viewable only in our platform. Providers can get access to detailed daily run data of our public benchmarks as well as early access to new benchmarks like this one. View our platform pricing.