robots.txt
A text file telling crawlers which parts of a site they can and can't access.
robots.txt is a plain-text file at /robots.txt that tells web crawlers (including AI crawlers like GPTBot and ClaudeBot) which paths they're allowed to access. It's the primary mechanism for controlling AI crawler access to a site.
If robots.txt blocks AI crawlers, the site won't appear in AI answers. As of 2026, roughly 18% of mid-market sites still block major AI crawlers by accident — often a legacy of the 2023 AI-training controversy.
See also