From OpenAI · · 1m 40s
OpenAI scales storage for 1 billion ChatGPT users
OpenAI uses Azure Cosmos DB to handle 20M QPS for 1 billion ChatGPT users.
In brief
OpenAI uses Azure Cosmos DB to handle 20M QPS for 1 billion ChatGPT users. OpenAI scaled Azure Cosmos DB to serve 20 million queries per second for over 1 billion ChatGPT users. Originally reported by OpenAI.
Habitat platform
OpenAI's online storage platform serves over 1 billion users across 40 geographic regions.
Scaling challenge
Habitat must respond in milliseconds to 1 billion users across 40 regions.
Phase I: Azure Cosmos DB
Initially used one default Azure Cosmos DB container per account.
Logical partition key limits
Collisions on user_id or workspace_id caused throughput constraints.
Noisy neighbors
One tenant consuming resources disrupted core login and signup flows.
Weak partitions
Physical partitions split as data grew, reducing effective throughput per partition.
Phase II: Single-tenancy
Created separate containers for critical use cases to isolate workloads.
Lack of stable growth
New services required 1,000x more headroom than legacy systems.
Capacity planning issues
Product teams forgot to pre-scale containers, causing launch delays.
Phase III: Multi-tenancy
Implemented co-managed containers with hot key throttling and qos.
Hot key throttling
Throttled hottest keys first to prevent noisy neighbor effects.
Quality of Service
Assigned lower priority to offline workloads to prevent throttling.
Serverless mode
Improved serverless mode allowed 10k ru/s per physical partition.
Reliability improvements
Joint engineering with Azure reduced outages for 10s of petabytes of data.
Global performance
Reduced global read p50 from 120 ms to 10 ms through caching and rust.
Future plans
Building custom exabyte-scale storage for next billion users.
The bottom line
OpenAI scaled Azure Cosmos DB to serve 20 million queries per second for over 1 billion ChatGPT users.
