Inside ChatGPT’s storage: OpenAI’s round trip on Cosmos DB tenancy

OpenAI published the second part of its account of Habitat, the storage platform behind its products. The company says Habitat serves over a billion users across 40 regions and holds more than 500 petabytes. Part I covered Habitat’s evolution from a Python client library into a service (Part I). Part II covers the Azure Cosmos DB layer: its partition model, where throughput per physical partition shrinks as data grows and partitions split, and a change-data-capture platform streaming millions of row changes per second through more than a hundred Kafka topics for caching, indexing and analytics. OpenAI says it runs what it believes is the largest Cosmos DB deployment anywhere, with tens of petabytes and more than 20 million queries per second on a single account. Every login, setting and piece of ChatGPT state runs through this layer.

OpenAI started with one shared container and hit noisy neighbors, where niche features disrupted core login and signup, so it moved critical work into dedicated containers. That cut blast radius but turned nearly every product team into capacity planners; teams forgot to pre-scale before launches and engineers ended up in incident rooms on launch day. OpenAI went back to a shared default with hot-key throttling, request priority tiers and a new serverless mode the Azure team built for it, and says it has had no provisioning or noisy-neighbor incident in the months since. The lesson is worth stealing: isolation has an organizational cost that never appears on an architecture diagram.

The latency work is solid: replicate only the smallest, most latency-sensitive data, such as authentication metadata, widely; use a Valkey cache near users with invalidations driven by the database change feed; and rewrite the proxy from Python into Rust. OpenAI also says that in regions short on power and cooling it sometimes chooses between storage near users and more serving compute. Physical capacity now shapes product latency.

In late 2024 a power outage hit the primary region for one of its largest accounts, and failover took hours, at a size neither side had tested. About a month later, failing back set off a connection stampede that overwhelmed the gateway. OpenAI’s conclusion is blunt: at its scale, managed services become less managed. The fixes were joint dashboards, a production-sized test account and direct paging between engineering teams. Unexplained simultaneous outages across AI providers show why this matters (outages).

That is the right fix for OpenAI and also a sign of concentration. The provider builds features for one customer and co-owns its failures, which works until incentives diverge, and smaller customers inherit tooling shaped around someone else’s edge cases. The closing note reads as a hedge: OpenAI is building exabyte-scale storage on RocksDB Cloud from Rockset with Raft replication and the same document model, keeping the Azure partnership while building the option to rely on it less. More in our OpenAI and Microsoft coverage.