Independent research on AI agent safety, crawler policies and what websites actually serve. Published by Emmanuel Orta, founder of CrawlCheck, with source code, dated datasets and reproducible evidence.
Follow the edition DOI to the archived files, then use the linked method or pinned source code to reproduce the work. For signed datasets, verify the edition and part hashes with the published signing keys and verification instructions. File checksums check transfer integrity. A signature identifies the signer and integrity of the record.
A frozen, reproducible test of whether agents stop before hazardous web actions, with and without an external verification signal.
What this record establishes
Models stopped on 32 of 80 hazard trials unaided and 79 of 80 with a correct external signal.
Scope and limits
39 operator-selected cases, four models, one run per case and model. The external signal was correct by construction. This measures whether models heed that signal, not how often the verification service is right.
Use this work
Re-run the matched hazard and control cases to compare agent behaviour under the published scoring rule.
Orta, E. (2026). Do AI Agents Stop Before Acting? A Frozen Benchmark of Lookalike Domains, Poisoned Tools, Revoked Receipts and Stale Answers, With and Without an External Verification Signal. Preprint. Zenodo. https://doi.org/10.5281/zenodo.23682818
How adding a named crawler group can drop restrictions inherited from the general robots.txt group.
What this record establishes
1,061 of 2,737 eligible sites (38.8%) gave at least one named crawler access to paths blocked by their general rules.
Scope and limits
One reading from one network, 26 crawler names, and 12,828 readable robots.txt files from a stratified 20,000-origin sample. Published rules do not establish operator intent or actual crawler behaviour.
Use this work
Reproduce the sample and parser comparison, or inspect a site for rules lost when a named group replaces the wildcard group.
Orta, E. (2026). Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites. Preprint. Zenodo. https://doi.org/10.5281/zenodo.23682816
A sealed weekly edition of crawler policies, machine files and declared capabilities on fixed top-site frames.
What this record establishes
Edition 2026-W44 preserves the weekly census separately from the changing live dashboard.
Scope and limits
Identity tests send crawler user-agent strings from CrawlCheck’s network. They describe the test client’s response, not independently authenticated traffic from those crawlers. Weekly comparisons use the published fixed frame.
Use this work
Compare sealed editions on the same frame and cite the exact weekly DOI when reporting a result.
Metadata captured from Zenodo. File checksums are repository-published MD5 values. Signed edition verification is a separate check.
Archived editions and live observations
The editions above preserve a dated result. Live figures carry their own source timestamps and can change. Use the exact edition DOI when citing a finding, and its series DOI when referring to the continuing dataset.
Collaborate or cover the work
Research replication, dataset reuse, interviews and partnerships are welcome. Include the publication DOI and the question you are exploring when you write to Emmanuel. Press kit · Press facts as JSON-LD.