ngrok vs PydanticAI Gateway Benchmark Comparison
Pydantic was faster in both shared AI gateway benchmarks.
- Shared benchmarks
- 2
- Last run
- October 30, 2026
Scorecard
Median of each benchmark's primary metric; lower is better. Medians within 5% count as roughly even, and a benchmark gets no speed verdict when either provider completed fewer than 50% of runs.
Anthropic
Run October 30, 2026Time for a cold request routed through the gateway to Claude Haiku 4.5 to return its complete response.
Cold end-to-end latency
Pydantic had a 1.15x lower median cold end-to-end latency (393ms vs 452ms for ngrok).
Median cold end-to-end latency over time
11 runs from August 21, 2026 to October 30, 2026.
OpenAI
Run October 30, 2026Time for a cold request routed through the gateway to GPT-5.4 Mini to return its complete response.
Cold end-to-end latency
Pydantic had a 1.11x lower median cold end-to-end latency (681ms vs 753ms for ngrok).
Median cold end-to-end latency over time
11 runs from August 21, 2026 to October 30, 2026.
Methodology
- One workload, every provider. The same script, payload, and model hit every provider, addressed the way its own API expects. Nobody gets a tuned or special path.
- Repeated on a schedule. Every suite runs many iterations per provider, automated on a recurring schedule, so results reflect steady state rather than one lucky request.
- Raw results published. Every run is committed as JSON to the public benchmarks repo, next to the code that produced it. Nothing is cherry-picked or held back.
- How this comparison is built. Each benchmark compares the two providers' medians; results within 5% count as roughly even. A comparison is published only when both providers completed the latency benchmark on at least two of the same model families reliably: at least 50% success for each provider, in benchmark runs from the last 30 days.
Frequently asked questions
Have another question? Email us.
Pydantic was faster in both shared AI gateway benchmarks. In Anthropic (October 30, 2026), Pydantic had a median cold end-to-end latency of 393ms versus 452ms for ngrok, 1.15x faster. In OpenAI (October 30, 2026), Pydantic had a median cold end-to-end latency of 681ms versus 753ms for ngrok, 1.11x faster.