Konstantin Ryabitsev, who runs kernel.org infrastructure for the Linux Foundation, published traffic figures on August 29, 2026 that are hard to argue with. The git.kernel.org web interface takes around six million requests a day for random commits. Two thirds are turned away at the proof of work challenge, a third solve it and carry on, and once you allow for every generous assumption, roughly two percent of what arrives is a person or a tool acting for one. Between fourteen and sixteen of the ninety cores in the fleet do nothing but render commit pages for crawlers. Here is what that costs, and what to do instead.
The short answer
Konstantin Ryabitsev published kernel.org traffic figures on August 29, 2026. The git web interface fields roughly six million requests a day for random commits. The Anubis proof of work gate turns away 66 percent of them outright, 33 percent solve the challenge and reach the site, and after generous assumptions only about 2 percent of the total looks legitimate. Between fourteen and sixteen of the ninety cores spread across five geographically distributed nodes are permanently occupied rendering commit pages for crawlers. Address based blocking does not help, because the traffic arrives through residential proxy networks. The next step is cutting the number of crawlable URLs.
Most of us guess at what share of our traffic is automated. Kernel.org measured it, and the answer is that the humans are a rounding error.
Six million requests for pages nobody reads
The number that matters is not the total, it is what the total is made of. Around six million requests a day arrive at git.kernel.org asking for individual commits, and they ask for different ones each time, which is the detail that turns an annoyance into a bill.
A commit page is not a file on disk. Serving one means reading git objects, computing a diff and rendering HTML. Ask for the same commit repeatedly and a cache absorbs it. Ask for six million different commits and there is nothing to cache, so every request is fresh work. Ryabitsev's accounting is that fourteen to sixteen of the ninety cores in the fleet, spread over five geographically distributed nodes, are doing nothing but that.
Two thirds of the arriving requests never get that far, because Anubis stops them at the door. The remaining third solve the challenge and go through. Even inside that third, once you make every assumption in favour of the visitor being real, the legitimate share of everything hitting the service comes to roughly two percent.
Why the usual defences do not apply
The instinct is to reach for a rate limit. It does not work here, and the reason is a business model rather than a technical trick.
There is a market for proxy software development kits: a maker of an application or a consumer device is paid to embed code that routes other people's traffic through the device's home connection. Smart televisions are among the hardware doing this. The buyer gets a very large pool of ordinary residential addresses, and the traffic that comes out of it looks exactly like a household. Each address sends a handful of requests. Blocking one is pointless. Blocking the range takes out real users on the same network.
That is what pushes an operator toward proof of work. Anubis does not try to decide who you are, it makes each request cost something, so the party generating volume pays for it. Kernel.org has already moved the difficulty from level 4 to level 5. The uncomfortable part of Ryabitsev's post is that a third of the traffic solves it anyway, because when the work is spread across a pool of that size, no single device feels the cost.
What to do if you need the data
If you are the one generating this load, there is a cheaper path for you too, and it is the same path that is cheaper for kernel.org.
Clone the tree once and fetch. Every commit is then local, diffing is free, and you have stopped asking a shared server to render something you could compute yourself. If you need a lot of repositories kept current, grokmirror was written for that job by the same person publishing these numbers. Mailing list history is in the public inbox archives on lore.kernel.org, which are git repositories you can clone rather than pages you have to walk. Give your crawler a user agent with a contact address in it. And if your case genuinely is not covered, ask, because the offer to hand over data on request is still open.
The part that generalises
Kernel.org is unusual in that it publishes its numbers. It is not unusual in the situation. Any service that exposes a large space of generated URLs, an issue tracker, a package index, a documentation site with per version permalinks, is looking at the same arithmetic, generally without the instrumentation to prove it.
The open source world has spent this year working out its terms with generative AI in public, from Debian voting on responsible use inside the project to public money going into Flatpak maintenance. This is the same argument arriving through the billing department. The next move at kernel.org is to reduce the number of crawlable URLs, which means the casual visitor loses a permalink so that the project can keep paying for the machines. That trade is worth naming, because a lot of infrastructure is about to make it.
Sources and further reading
- Creepy crawlies, Konstantin Ryabitsev, people.kernel.org, August 29, 2026
- Ryabitsev: Creepy crawlies, LWN.net, August 29, 2026
- Anubis, the proof of work gateway used by kernel.org
- grokmirror, the kernel.org repository mirroring tool
Frequently asked questions
What is Anubis and why is kernel.org using it?
Anubis is a proof of work gate that sits in front of a web application. Before serving the page it asks the client to find a string which, combined with the client address and a secret the server provides, hashes to a SHA-256 sum with a set number of leading zeroes. A browser does that in a fraction of a second and moves on. A crawler doing it millions of times pays real processor time, which is the point: the cost lands on whoever is generating the volume. Kernel.org has already raised the difficulty from level 4 to level 5, which multiplies the work required for each pass.
Why does crawling a git web interface cost so much?
Because a commit page is generated, not served from disk. Every request for a random commit makes the server read objects, compute a diff and render it as HTML. One visitor doing that is nothing. Millions of requests a day, each asking for a different commit so nothing stays usefully cached, turns into sustained compute. Ryabitsev puts it at fourteen to sixteen cores out of ninety across five geographically distributed nodes doing nothing else, which is capacity that was bought and is being consumed to produce pages no human will read.
Why do rate limits and IP blocks not solve it?
Because the requests do not come from a small set of addresses. A market exists for proxy software development kits that pay app and device makers to route third party traffic through their users' home connections, including consumer devices such as smart televisions. The result is a very large pool of ordinary residential addresses, each sending a small number of requests. Blocking any one of them achieves nothing, and blocking the range they sit in would take out real users on the same network. That is what pushed kernel.org toward a cost based challenge instead of an address based rule.
I actually need kernel history at scale. What should I do?
Clone rather than crawl. A single git clone of the tree gives you every commit locally, and git fetch keeps it current for a tiny fraction of the traffic that walking commit URLs generates. If you need many repositories mirrored and kept in sync, grokmirror exists for exactly that job and was written by the same person publishing these numbers. Mailing list history lives in public inbox archives on lore.kernel.org, which are themselves git repositories you can clone. And if none of that covers your case, ask: Ryabitsev has been explicit that the data stays available to people who request it.
What is kernel.org changing?
The immediate step is reducing the number of crawlable URLs, which in practice means turning off features of the web interface that generate large numbers of distinct pages. That is a loss for the casual browser who wanted a permalink to a diff. It is also the only lever that works without spending more money on hardware, because the underlying problem is not any single crawler but the sheer number of distinct pages a commit browser can be asked to produce.