Can AI read your website?

Can ChatGPT see my website?

It can if three things are true: your robots.txt allows OpenAI’s crawlers, your firewall or host lets them in, and the page’s text is in the HTML your server sends. The same goes for Claude and Perplexity, which have crawlers of their own.

Google’s AI Overviews and AI Mode, and Microsoft’s Copilot, use the search engines’ own crawlers. Google says its robots.txt rules for Googlebot control access for its AI features, and Bing says Copilot relies on the same crawling and indexing as Bing search. None of this makes an AI tool mention you, but a page its crawlers can’t read is one it can’t draw on.

What are GPTBot, OAI-SearchBot and the other AI crawlers?

They’re the names AI companies’ crawlers go by in robots.txt. Each company runs separate crawlers for separate jobs, so you can allow one and block another – OpenAI says each setting is independent of the others.

Sources: OpenAI, Anthropic, Perplexity, Google and Microsoft, read in October 2026. Your results say when we last checked this list.

How do robots.txt rules apply to AI crawlers?

A crawler follows the group of rules that names it and ignores the rest; if no group names it, it follows User-agent: *. The longest matching rule wins, under both the robots.txt standard, RFC 9309, and Google’s rules. In practice:

• Disallow: / under User-agent: * blocks every crawler without a group of its own, Googlebot and Bingbot included.
• A crawler with its own group ignores the * rules. If you add User-agent: OAI-SearchBot and Allow: /, repeat any * rules it should still keep to.
• No robots.txt is fine. A “not found” answer lets crawlers read every page.
• A server error isn’t. Crawlers that follow the standard then assume they may read nothing, and Google stops crawling the site for the first 12 hours.

Changes can take up to a day to be noticed: Google generally caches robots.txt for up to 24 hours, and OpenAI and Perplexity give similar times.

Should I block AI crawlers?

That’s your choice, but decide by job, not by company. Blocking the training crawlers – GPTBot, ClaudeBot and Google-Extended – asks them not to train on your pages, and doesn’t take you out of ChatGPT, Claude or Perplexity search, or Google’s AI Overviews, so our checker treats it as a choice, not a fault. The one catch is Google-Extended, which also covers Gemini using your pages to ground its answers – blocking it asks Google to keep them out of those too. Blocking the AI search or on-request crawlers asks those tools not to read the page when they answer.

Don’t block Googlebot to stay out of AI Overviews: that stops Google crawling the page for Search too. Google’s nosnippet rule keeps a page from being a direct input for AI Overviews and AI Mode, but also removes the text under its link in ordinary results. For Microsoft’s Copilot, the rule is noarchive, which Bing says stops Copilot using the page in its answers or linking to it. Our checker looks for both.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot collects pages that may be used to train OpenAI’s models; ChatGPT search uses OAI-SearchBot. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links. It’s the same elsewhere: a rule for ClaudeBot doesn’t cover Claude-SearchBot or Claude-User, and Google says Google-Extended doesn’t affect a site’s inclusion or ranking in Google Search.

Why can’t AI read my JavaScript website?

Because some of the main AI crawlers have been found not to run JavaScript, so they see only the HTML your server sends. In Vercel’s tests, published in December 2024, OpenAI’s crawlers, Anthropic’s ClaudeBot and PerplexityBot didn’t run it, though Apple’s and Google’s did. Anthropic’s newer Claude-SearchBot and Claude-User, and Perplexity-User, weren’t tested, and none of the three companies documents it either way. Googlebot does run JavaScript, though Google queues pages to do it and notes that not all bots can.

So a JavaScript app can reach those crawlers as a near-empty page, and React and Next.js can stream a page’s text in hidden containers behind a loading screen. The fix is server-side rendering or pre-rendering – in Next.js, often removing a loading.js file or Suspense boundary around the whole page. View Source in your browser shows the HTML your server sends.

Is robots.txt enough to let AI crawlers in?

Not always. robots.txt is a request, not a lock – Google notes it can’t enforce what crawlers do – and a firewall, CDN or host’s security can turn a crawler away whatever it says, which our checker can’t see. Google’s AI features guidance says to check that your CDN and hosting allow crawling too. Cloudflare’s AI Crawl Control, for example, can block individual AI crawlers with a firewall rule.

Equally, if your site’s security stopped or refused our checker’s request for robots.txt, that doesn’t mean AI crawlers are blocked – or that they’re allowed in – so we don’t judge them then.

How do I edit robots.txt in WordPress, Squarespace, Wix or Shopify?

• WordPress with Yoast SEO – Yoast SEO › Tools › File editor, shown only if your site allows file editing.
• WordPress with Rank Math – turn on Advanced Mode, then Rank Math SEO › General Settings › Edit robots.txt, once any robots.txt file in your site’s root folder is removed.
• Squarespace – Settings › Crawlers has a “Block known artificial intelligence crawlers” box, off by default. It covers GPTBot, ClaudeBot, Google-Extended and 23 others, but not OAI-SearchBot, Claude-SearchBot or PerplexityBot.
• Wix – SEO & GEO › Tools and settings › Robots.txt Editor, then View File.
• Shopify – choose Edit code on your theme and add a robots.txt.liquid template. Shopify warns that a mistake can lose you all your traffic, and suggests working with a Shopify Partner.

Reading your results

• Fail – robots.txt blocks Googlebot or Bingbot from the page, or a noindex asks search engines not to list it. Either keeps the page’s content out of search, and so out of the AI answers built on it.
• Warning – robots.txt blocks an AI search or on-request crawler, or answers with a server error; a nosnippet rule, or Bing’s noarchive; an X-Robots-Tag header that may say noindex; or text that needs JavaScript or mostly arrives hidden.
• Note – worth knowing: whether AI training crawlers are allowed; no robots.txt, or one we couldn’t read; Bing’s nocache; little text, or some arriving hidden; what we found about your business; and, every time, the firewall reminder.
• Pass – Google, Bing and the AI search crawlers may read the page, nothing asks search engines to leave it out, its text is in the HTML, and structured data names your business.

The crawler table marks each crawler Allowed or Blocked by the rules for that page’s address, with the rule that decides it. When we couldn’t read robots.txt, there’s no table, just a line saying why. Pasted HTML skips robots.txt and the X-Robots-Tag header. There’s no score.

Common questions

Do I need an llms.txt file?

No. llms.txt is a proposal, first published in 2024, for a Markdown file that guides AI tools to a site’s key pages. Google says you don’t need AI text files to appear in AI Overviews or AI Mode. Our checker doesn’t look for one, and we can’t promise one would get you mentioned.

Does structured data help AI tools find my business?

It can help search engines understand who you are, but guarantees nothing. Google says Organization structured data helps it understand your business’s details, though its AI features need no special markup; Bing says structured data doesn’t guarantee visibility. Our checker reports what yours says, and whether the page’s text has a UK postcode or phone number.

Does the checker test my whole site?

No. It checks one page and the robots.txt rules for its address, which can differ from one section of a site to another. Start with your home page, then a page from each important section. Each connection can run 15 checks a day.

What do you keep when I run a check?

Not your results. Pasted HTML is checked in memory and never stored or logged, and nothing is fetched for it. For an address, we also read the site’s robots.txt – a public file we may reuse for up to 10 minutes, so repeat checks don’t ask the site for it again. For 7 days we keep a short log of each check: the tool, the domain, the outcome, how long it took and a code for your connection that changes daily. We never store your IP address, and the counts behind the daily limits are deleted after 48 hours. Cloudflare, which runs the checker and its Turnstile bot check, does process your IP address. Our privacy policy and About our checker have the details.