KONSTANTIN Ryabitsev, head of IT infrastructure at the Linux Foundation, reports that AI web crawlers are straining the server resources of git.kernel.org significantly. Out of approximately 6 million daily requests for Git commits, about 66% are rejected by the Anubis proof-of-work mechanism designed to filter out bots; however, 33% still succeed.
Estimates suggest that only about 2% of the traffic is from legitimate users, with crawlers consuming 20% of CPU capacity by rendering HTML for 1.48 million commits, rather than using more efficient Git cloning. The crawlers are now bypassing IP bans by mimicking regular browser traffic, which complicates enforcement of anti-scraping measures.
Ryabitsev indicates future actions may include limiting scrapable URLs and imposing tighter anonymous access, highlighting the inefficient burden on open-source infrastructure.