Preprint ·
Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites
How adding a named crawler group can drop restrictions inherited from the general robots.txt group.
What this record establishes
1,061 of 2,737 eligible sites (38.8%) gave at least one named crawler access to paths blocked by their general rules.
Scope and limits
One reading from one network, 26 crawler names, and 12,828 readable robots.txt files from a stratified 20,000-origin sample. Published rules do not establish operator intent or actual crawler behaviour.
Use this work
Reproduce the sample and parser comparison, or inspect a site for rules lost when a named group replaces the wildcard group.
Download files (2)
- robots-shadowing.tex · 18,252 bytes
md5:b678066be9c617d0fc641e0cf59855a0 - robots-shadowing.pdf · 171,185 bytes
md5:62b8a9fe809a3dbd2efeea58ef44d7ae
Cite this edition
Orta, E. (2026). Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites. Preprint. Zenodo. https://doi.org/10.5281/zenodo.23682816
BibTeX · RIS · CC BY 4.0 reuse terms
Metadata captured from Zenodo. File checksums are repository-published MD5 values. Signed edition verification is a separate check.