An AI crawler is a program that AI companies use to read websites, such as GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended.
If your robots.txt file blocks these crawlers, the AI tools they feed can’t read your pages, however good the pages are.
A web designer adds “Disallow: /” for GPTBot by mistake. ChatGPT search stops seeing the site’s new service pages.
Start with the free score, or book a 30-minute call and we will walk through it with you.
The score is free, no account needed. The call is 30 minutes.