robots.txt is one of the fastest ways to accidentally make your website invisible to AI answer engines. This landing page explains how robots rules apply to AI crawlers and what to check for when visibility is the goal. Use the free homepage audit to generate a report that includes detailed bot-by-bot access results.
robots.txt is a rules file that tells crawlers which paths they can access. Many AI engines rely on crawler access to fetch pages for citations and summaries. If robots.txt blocks key AI bots, your content may not be visible to those systems.
If your goal is AI visibility, you typically want to allow citation-focused crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Some sites still choose to block training-focused bots. The right choice depends on your content and business goals.
You can review your robots.txt for wildcard rules like `User-agent: *` with `Disallow: /`, and for bot-specific groups. This checker audits the most common AI crawlers and tells you if they are allowed, blocked, or blocked by wildcard rules. Always retest after editing the file.
Allowing all bots can increase discovery, but it can also increase crawling load and may not align with your content policies. A more controlled approach is to allow citation-focused bots while limiting training bots if desired. You should also ensure your server can handle crawl traffic.
If ChatGPT-related crawlers are blocked, your pages may not be fetched for citations or live retrieval. That can reduce the chance your site appears as a source in AI answers. Content quality still matters, but access is a prerequisite for being referenced.
