LLM Gateway vs PydanticAI Gateway Benchmark Comparison
Pydantic was faster in both shared AI gateway benchmarks.
- Shared benchmarks
- 2
- Last run
- October 30, 2026
Scorecard
Median of each benchmark's primary metric; lower is better. Medians within 5% count as roughly even, and a benchmark gets no speed verdict when either provider completed fewer than 50% of runs.
Anthropic
Run October 30, 2026Time for a cold request routed through the gateway to Claude Haiku 4.5 to return its complete response.
Cold end-to-end latency
Pydantic had a 1.41x lower median cold end-to-end latency (393ms vs 553ms for LLM Gateway).
Median cold end-to-end latency over time
14 runs from July 24, 2026 to October 30, 2026.
OpenAI
Run October 30, 2026Time for a cold request routed through the gateway to GPT-5.4 Mini to return its complete response.
Cold end-to-end latency
Pydantic had a 1.75x lower median cold end-to-end latency (681ms vs 1191ms for LLM Gateway).
Median cold end-to-end latency over time
10 runs from August 21, 2026 to October 30, 2026.
Methodology
- One workload, every provider. The same script, payload, and model hit every provider, addressed the way its own API expects. Nobody gets a tuned or special path.
- Repeated on a schedule. Every suite runs many iterations per provider, automated on a recurring schedule, so results reflect steady state rather than one lucky request.
- Raw results published. Every run is committed as JSON to the public benchmarks repo, next to the code that produced it. Nothing is cherry-picked or held back.
- How this comparison is built. Each benchmark compares the two providers' medians; results within 5% count as roughly even. A comparison is published only when both providers completed the latency benchmark on at least two of the same model families reliably: at least 50% success for each provider, in benchmark runs from the last 30 days.
Frequently asked questions
Have another question? Email us.
Pydantic was faster in both shared AI gateway benchmarks. In Anthropic (October 30, 2026), Pydantic had a median cold end-to-end latency of 393ms versus 553ms for LLM Gateway, 1.41x faster. In OpenAI (October 30, 2026), Pydantic had a median cold end-to-end latency of 681ms versus 1191ms for LLM Gateway, 1.75x faster.