# Five years of petabyte ClickHouse, and what broke > Tinybird's co-founder on the parts of ClickHouse that fail quietly at scale, and the habits that keep a petabyte cluster alive. - Title: Five years of petabyte ClickHouse, and what broke · Tinybird - Summary: Tinybird's co-founder on the parts of ClickHouse that fail quietly at scale, and the habits that keep a petabyte cluster alive. Standing up ClickHouse takes… - Keywords: clickhouse, databases, olap, infrastructure, scaling, technology, startups, Five, years, petabyte, broke, Tinybird - Source: Tinybird — https://tinybird.co/blog/what-i-learned-operating-clickhouse - Read time: 2 min - Topics: clickhouse, databases, olap, infrastructure, scaling, technology, startups ## Six years, one cluster Javi Santana has run ClickHouse at Tinybird since version 18.4, across many petabyte-scale clusters. His summary after all that time is blunt about where the difficulty actually sits. > Setting up a cluster is easy, the hard part is keeping it running. ## Replicas first, shards later Tinybird ran with no shards at all: scale replicas vertically for bigger queries, add replicas for more traffic. Careful schema design let them put sharding off for years. > re-sharding was super hard ## The load balancer is the brain An HTTP load balancer routes each request by load, replica type and consistent hashing, so cache hits land on the same machine. They skip the native TCP protocol to keep standard HTTP tooling. > The load balancer is the key to all of this. Any modern LB would do the job. ## Give writes their own replica Separate the replica that ingests from the ones that serve reads, with automatic failover. If a query class has a p99 target, isolate it on a replica held under 40 percent load. > Real-time sub-second queries are expensive; forget about those cheap batch jobs running on spot instances. ## The replication math gets ugly A 300TB table at 1000 QPS, with each replica good for 100 QPS, means ten full copies and 3000TB of disk. On SSDs, that bill is the whole argument for object storage. ## Cloud storage is the weak spot Zero-copy replication onto S3 came from an outside contributor, and ClickHouse Inc. would rather it went away. It works if you watch it, but part operations can quietly lose data. > it's buggy, you can lose data, it leaves garbage in S3 ## Hot SSD, cold S3 Tinybird caches on local SSD in front of S3 for latency-sensitive customers, uses compact parts to cut S3 write ops, and picks ZSTD(1) over LZ4 as the default compression. > This approach provides the required low latency that S3 alone cannot deliver. ## Upgrade one replica at a time Add a replica on the new version, read its logs, send it read traffic, then writes, then real traffic, then move to the next. The first upgrade took three hours and two weeks of prep. > It only took us 4 years to figure out. AFAIK, there is no other company doing this. ## Wait a month after a release New versions shift on-disk formats, SQL behavior, performance and default settings. Replay your real customer queries against the next version daily, and stay away from experimental features. > Do not upgrade anything right away after the release; wait at least a month. ## You will read the source Running ClickHouse well means reading its code, something Santana never needed across hundreds of Postgres clusters. Most of the biggest users carry patches upstream. > Do not test a single instance; test against a cluster with at least two machines and the Zookeeper or ClickHouse® Keeper. ## Budget for ZooKeeper and people Three ZooKeeper replicas, on their own machines. Share a box and one overload drops your tables to read-only. Rule of thumb for sizing: 32 cores handles about 5GB/s of uncompressed data. > if you are not paying US salaries, having someone looking after the cluster pays off ## Ingestion is where data dies Inserts create parts, merges combine them. Batch too small and the merge queue drowns; batch too big and inserts run out of memory. An OOM mid-ingest means rows you never get back. > Every single company handling ClickHouse® struggles with ingestion. ## Key takeaway Standing up ClickHouse takes an afternoon. Keeping it up means owning ingestion, upgrades and storage costs before they own you.