Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers

Jul 17, 2026

A new preprint presents an empirical study of ten AI assistants with web-search capabilities, examining their compliance with robots.txt website restrictions. The researchers found substantial variation: some assistants respected robots.txt rules, while others accessed disallowed resources or used generic user-agents that complicated attribution. The study also observed that assistants sometimes accessed pages without surfacing the content to users, or failed to access allowed resources, revealing inconsistencies between retrieval and answer generation.

Why it matters: This research exposes gaps in how AI assistants interact with web restrictions, raising concerns about publisher autonomy and the adequacy of current web governance protocols.

Full story at: arXiv Computers and Society ↗