CLOUDFLARE reports that more than 20% of the web sits behind its network, and the Data team runs the data pipeline powering Logs, SQL API, observability and more. Faced with traffic growing at exponential rates against linear hardware growth, the team describes a sequence of architectural and algorithmic changes since 2022 to avoid a proportional hardware ramp.
Key steps included breaking up a monolithic data schema in 2022, adopting zstd compression in 2024 (after an initial period of higher CPU usage), and rolling out a revamped load of analytics plumbing in 2026 to push data processing closer to the source and reduce throughput on the backbone.
The net effect has been substantial headroom gains: about 70 GB/s of compressed traffic eliminated across the backbone, with the uncompressed bytes continuing to grow but the system now able to handle more traffic with the same hardware footprint.
In 2026 Cloudflare implemented three major strategies to cut uncompressed bytes and improve resilience. First, a Shared Ingest Pool replaced dual-writing to two regions, so each event is transmitted once and then sampled or aggregated downstream. Second, late replication moves most of the cross-data-centre work to Buftee and downstream inserters, dramatically reducing cross-region traffic.
Third, early sampling and a new aggregation capability fold processing earlier in the pipeline, cutting both the volume of data sent and the number of rows stored in ClickHouse. Early results show sharp reductions, notably around Ready-Analytics and HTTP rollups, with multi-dataset cuts in the tens of GB/s range and reported reductions of up to around 50% for some datasets.
By keeping data processing efficient, Cloudflare says throughput is now roughly eight months behind its previous growth trajectory, with long-term impact expected to halve five-year data volumes.