CLOUDFLARE’S latest post argues that cache design must adapt to the AI era, where AI crawlers now account for a substantial portion of automated traffic and alter how content is consumed. Cloudflare data shows that 32% of traffic across their network originates from automated traffic, including AI assistants using retrieval-augmented generation.
AI crawlers exhibit high unique URL ratios, content diversity, and crawling inefficiency, which can drive up cache misses and reduce hit rates, undermining traditional strategies like prefetching and LRU-based eviction. The article cites real-world impacts, noting that Wikimedia experienced a 50% surge in multimedia bandwidth usage due to bulk image scraping, while Fedora and the Diaspora network faced slowdowns and heavy load for human users.
In response, the authors describe AI-aware caching approaches developed with ETH Zurich, including traffic filtering and the possibility of a separate cache layer dedicated to AI traffic, plus exploration of new algorithms such as SEIVE or S3FIFO to preserve human performance. According to ETH Zurich, long-term cache architectures may route human and AI traffic to distinct tiers to improve efficiency for all users.