Cloudflare Frees 100TB of RAM by Reworking Its Load Balancer

Cloudflare says it reclaimed more than 100TB of RAM globally by redesigning memory use in Pingora Backend Router (PBR), its internal load-balancing service. The problem was traced to pingora-ketama, an open-source consistent-hashing library used to route cacheable requests to suitable servers. PBR was using up to 6GB in some cases because it stored large numbers of hash points across many rings, including separate rings needed for combinations of compliance and caching features.

One change reduced the memory required for each hash point by 25%. The original Rust structure used a 32-bit hash and a 32-bit server index; Cloudflare said PBR was unlikely to coordinate more than 2^16, or about 65,000, servers, so the index could be represented with 16 bits. Because Rust’s alignment rules would otherwise retain an eight-byte structure, the team stored both values in a six-byte byte array.

Mathematical analysis also showed that PBR could reduce the number of hashes generated per server by 90% without appreciable distribution error. Cloudflare noted that adding hashes offers diminishing returns and that collisions become more likely with 32-bit hashes, particularly between 10,000 and 100,000 hashes per server in simulations involving 2,048 servers.

To avoid widespread cache disruption and a surge in origin traffic, Cloudflare ran the old and new rings simultaneously, gradually migrated traffic by data centre, monitored cache and error metrics, and retained a rollback path. After the migration reached 100%, the old implementation was removed. The changes are available in pingora-ketama through an, for now, unadvertised Cargo feature supporting the new “v2” ring.