Kernel.org administrator Konstantin Ryabitsev estimated that only about 2% of git.kernel.org requests were legitimate. The qualifier matters: this was an operational estimate under explicit uncertainty, not an audited count of human visitors.
Three denominators to keep separate
In Creepy crawlies, Ryabitsev describes about six million daily requests, with roughly 66% rejected at the challenge and 33% getting through. He explicitly says reliably separating bots from humans is difficult. Solving the challenge is not proof that a person is browsing. The rounded percentages also should not be forced into an exact 100% census.
His roughly 2% figure concerns requests judged legitimate, not the proportion of accounts, unique people or all traffic across every kernel.org service. Our earlier headline erased those distinctions.
Convert capacity carefully
The post reports 14 to 16 cores rendering commits for scrapers out of 90 cores across five nodes, and describes the burden as around 20% with spikes. The direct arithmetic is 14 / 90 = 15.6% and 16 / 90 = 17.8%. The chart shows that interval and labels its denominator. It does not turn a rough prose estimate into a precise independent measurement.
Request share and processing cost need not match. A cached small response and an expensive generated commit page each count as one request, while consuming different amounts of CPU. A blocked request can still cost edge resources without reaching the origin.
Diagnose your own service by path
For a useful local audit, group requests by endpoint, response size, latency and cache outcome. Look for concentration on expensive URLs and simultaneous bursts, not merely a suspicious user-agent string. Compare origin CPU during the same intervals. A high request count with low origin cost is a different incident from a few costly queries.
Do not treat passing a browser challenge as a user identity signal. Likewise, an unfamiliar address or automation client can be legitimate. Evaluate the operational effect of a control: reduced origin work, acceptable access for intended users and a supported machine-readable route for legitimate consumers.
Avoid repeatedly rebuilding the same Git view
Ryabitsev also distinguishes scraper load from poorly designed CI that performs concurrent shallow clones. The grokmirror project provides repository-mirroring tooling. For CI, a maintained local mirror can reduce repeated remote fetches, but it still needs appropriate update handling, storage and repository integrity checks.
Before changing a pipeline, measure its current clone pattern and test a mirror on a disposable job. Verify the requested commit is present and build from that exact revision. Do not replace an expensive fetch with an unverified stale checkout.
Sources
Restored uncertainty around the administrator’s traffic estimate; separated challenge outcomes, request shares and CPU capacity, with corrected 14-16/90 arithmetic.