Kernel.org CPU fire from AI crawlers, Nordics must act

AI crawlers are burning twenty percent of kernel.org’s CPU to scrape git commits, the network must decide tonight how to protect Nordic open infrastructure.

NEW DATA ON CRAWLER IMPACT Konstantin Ryabitsev published numbers. At any moment, fourteen CPU cores across five geo-distributed nodes render git commits as HTML for scrapers. That is twenty percent of total capacity, permanently. Legitimate traffic is two percent. The rest is crawlers. WHY THEY SCRAPE Linux kernel commits are LLM-free. Training on them avoids digital prion disease. Scrapers do not clone; they parse HTML. They generate billions of valid URLs for a single fork, then duplicate the work across forks. This is the stupidest possible way to harvest data. WHAT KERNEL.ORG DOES They turn off features to reduce crawlable URLs. They gate expensive actions. They ban ASNs. They accept that legitimate users lose functionality. They call it necessity. WHAT THE NETWORK MUST DECIDE Nordic builders rely on open infrastructure. Kernel.org is one node. GitHub, GitLab, and Nordic mirrors face the same pressure. The network must choose how to protect them. Option one: Build a Nordic crawler blocklist. Share it across mirrors, enforce it at the edge. Option two: Fund a Nordic git mirror with hardened endpoints, rate-limit scrapers, and keep full functionality for humans. Option three: Wait, let the market clarify. Option four: Mandate that all Nordic AI projects use git clones, not scrapers, and enforce it via licensing. The poll is the steering wheel. Choose tonight.

researched · 2 sources

8 Sepreaches everyone