Who is reading your website now? A plain guide to AI crawlers

A year ago the question was whether Google could read your website. Now there’s a second audience: ChatGPT, Claude, Gemini and Perplexity, which answer questions about your brand whether or not you’ve thought about them.

Most of the confusion I see comes from treating “AI bots” as one thing. They’re not. Each company sends separate crawlers for separate jobs, and the difference decides whether you show up in their answers.

Three jobs, three kinds of bot

• Training. Collects pages that may be used to train future models. OpenAI’s is GPTBot and Anthropic’s is ClaudeBot. For Google, the Google-Extended setting controls whether your content is used for Gemini.
• Search. Indexes pages so the assistant can find and cite them when it searches the web: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity.
• Live visits. Opens a page in the moment, because a user asked about it: ChatGPT-User and Claude-User.

They work independently. OpenAI’s documentation is explicit that blocking GPTBot only opts you out of training, while sites that block OAI-SearchBot aren’t shown in ChatGPT search answers. Anthropic says the same about Claude-SearchBot.

The mistake that makes brands invisible

Plenty of sites block every AI bot with one line in robots.txt, often because a plugin, an agency or a hosting setting did it for them. Cloudflare, for example, started blocking AI crawlers by default for new customers in July 2025. That’s a reasonable choice for training. For search and live visits, it means that when an assistant is asked about your brand, it has to answer from whatever other sites say about you.

My default for a brand that wants to be found: decide about training separately, but allow the search and live-visit bots. Then check both your robots.txt and your firewall settings, because they don’t always agree.

What about llms.txt?

llms.txt is a proposed file that summarizes a website for language models. Adoption has grown fast, to about 36,000 sites by May 2026, but Ahrefs found that 97% of those files got no requests at all that month. Google says it isn’t needed for Search. It’s harmless to add. It’s not where your visibility comes from.

Google’s AI answers need nothing special

For AI Overviews and AI Mode, Google states there are no additional requirements: no new files and no special markup. If a page is indexed and eligible to show with a snippet in regular Search, it’s eligible there too. Blocking Google-Extended doesn’t remove you from Search.

What actually helps

• Pages bots can read. Your core facts in plain HTML text, not only inside images, sliders or scripts.
• An About page that states the facts. Who you are, where you are, since when, what you make and who it’s for. Assistants repeat what’s stated clearly.
• One name, used consistently. The same spelling of your brand across your site, your Google profile and your social accounts.
• Structured data that matches the page. Company details, products and FAQs that describe what visitors actually see.
• Other people writing about you. Press, directories and partner pages are what assistants lean on when your own site is silent.

If you want to see where your site stands, my free AI Visibility Check tests crawler access, structured data and your About and FAQ pages, and shows what an AI assistant says about your brand from memory. It takes about a minute.