Emmanuel Orta · VSNARY · Open research

Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites

How adding a named crawler group can drop restrictions inherited from the general robots.txt group.

ORCID · Research CV · Machine-readable catalogue · All BibTeX · All RIS

All research and dataset editions

Trace a result to its evidence

Follow the edition DOI to the archived files, then use the linked method or pinned source code to reproduce the work. For signed datasets, verify the edition and part hashes with the published signing keys and verification instructions. File checksums check transfer integrity. A signature identifies the signer and integrity of the record.

Publishing and measurement method · Corrections · Explore all CrawlCheck datasets

Preprint ·

Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites

How adding a named crawler group can drop restrictions inherited from the general robots.txt group.

What this record establishes

1,061 of 2,737 eligible sites (38.8%) gave at least one named crawler access to paths blocked by their general rules.

Scope and limits

One reading from one network, 26 crawler names, and 12,828 readable robots.txt files from a stratified 20,000-origin sample. Published rules do not establish operator intent or actual crawler behaviour.

Use this work

Reproduce the sample and parser comparison, or inspect a site for rules lost when a named group replaces the wildcard group.

Download files (2)

Cite this edition

Orta, E. (2026). Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites. Preprint. Zenodo. https://doi.org/10.5281/zenodo.23682816

BibTeX · RIS · CC BY 4.0 reuse terms

Metadata captured from Zenodo. File checksums are repository-published MD5 values. Signed edition verification is a separate check.

Archived editions and live observations

The editions above preserve a dated result. Live figures carry their own source timestamps and can change. Use the exact edition DOI when citing a finding, and its series DOI when referring to the continuing dataset.

Collaborate or cover the work

Research replication, dataset reuse, interviews and partnerships are welcome. Include the publication DOI and the question you are exploring when you write to Emmanuel. Press kit · Press facts as JSON-LD.