From Mmoustafa · Mo Moustafa · · 2m
So you want to use OpenRouter?
18 million messages through OpenRouter taught Mo Moustafa that the same model is a different model depending on who hosts it.
In brief
18 million messages through OpenRouter taught Mo Moustafa that the same model is a different model depending on who hosts it. On OpenRouter you are not buying a model, you are buying a provider. Benchmark them yourself, from prod, and never pin your way into a single point of failure. Originally reported by Mmoustafa.
The model is not the product
Ask for deepseek/deepseek-v4-flash and you get one of about 20 hosts you've never heard of. Same weights, their GPUs, their precision, their parsers, their bugs.
“They're the same model on paper, but very different models in real life.”
Same weights, 20 point swing
On DeepSeek V4 Flash, first-party scored 90% GPQA and 81% TAU. DigitalOcean served the identical weights at 75% and 58%.
“For an agent TAU is the score that matters and a 20 point swing is not noise.”
Some vision providers are blind
DeepInfra's Qwen endpoint read a K as an R and called red blue. Venice and Together never saw the MiniMax images at all, and returned 200 OK anyway.
“The model page says it supports image input, but two of its providers don't”
The effort knob may do nothing
reasoning.effort is accepted everywhere but honored selectively. Testing low, high and max across every DeepSeek host showed four providers whose reasoning token counts barely moved.
Precision is a bad proxy
Filtering for fp8 over fp4 buys you nothing. The fp4 hosts landed mid-pack, and GLM's top scorer declared no precision at all while shrinking your fallback pool.
“Filter on the board, not the bits.”
Tool calls leak into text
Provider parsers miss, and the raw markup lands in the reply as user-facing content. You will end up parsing wrapped tool calls and half-wrapped responses yourself.
“<use_skills><parameters>{"skills":["search"]}</parameters></use_skills>”
200 OK with no answer
Reasoning models dump everything into the reasoning field and return content: null with finish_reason stop. 345 tokens billed, nothing to show the user.
“A 200 tells you the request was served, not that there's an answer in it.”
Hollow completions happen at scale
Some endpoints return null content, null reasoning, and no usage object. StreamLake was 20% of traffic in July and 92% of the empty completions.
The contract is per provider
SiliconFlow 400s with code 20015 if you pass back empty reasoning_content from a tool call. Baidu, Alibaba and Cloudflare accept the exact same history.
“So the contract isn't per model, it's per provider.”
Pinning is not safety
Three reliable providers pinned with fallbacks off. Two weeks later Baidu was 429ing, Cloudflare had dropped the model, and Alibaba buckled under 100% of traffic. Olly went down.
“Benchmark from where prod runs, a few at a time, more samples than feels necessary.”
The main idea
On OpenRouter you are not buying a model, you are buying a provider. Benchmark them yourself, from prod, and never pin your way into a single point of failure.
