AEOAgent.Ai

robots.txt AI Crawler Checker

Paste your robots.txt below to see which AI crawlers can access your site. We check for 12 AI bots including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

How to find your robots.txt

  1. Visit yourdomain.com/robots.txt in your browser
  2. Select all text and copy it
  3. Paste it above and click "Check AI crawler access"
  4. Review which bots are blocked
  5. Run an AEO scan to verify your AI visibility

Why AI crawler access matters for AEO

AI answer engines can only cite your content if their crawlers can reach it. A single Disallow: / directive for GPTBot means ChatGPT — with 400+ million users — cannot reference your pages.

Allow these six critical bots: GPTBot, ChatGPT-User, OAI-SearchBot (OpenAI), Google-Extended (Gemini), ClaudeBot (Anthropic), and PerplexityBot. Together they drive 95%+ of AI search traffic.

AI crawler access is 1 of 12 AEO factors. Get your full score.

Run free AEO scan