Blog

The technical side of AI search: crawlers, schema and llms.txt

Which AI bots visit your site, what to let them do and which technical measures for AI search engines actually make a difference.

Key takeaways

  • AI crawlers fall into three groups: bots for training, bots for a search index and agents that fetch a page live when a user asks something.
  • In robots.txt you can block training while allowing search, for example by disallowing GPTBot and allowing OAI-SearchBot.
  • Google-Extended isn't a crawler but a robots.txt token, and it has no effect on Google Search or AI Overviews.
  • Google says structured data isn't required for AI features in Search, but it remains useful for rich results and for clarifying who you are.
  • Google Search ignores llms.txt; the file is harmless and may help other services, but it isn't a ranking factor.

Technically, AI search engines need much the same as Google: your content must be crawlable, indexable and readable without JavaScript, and robots.txt lets you decide which AI crawlers may do what. Structured data and llms.txt are aids, not an entry ticket. Google states explicitly that neither is needed to appear in AI Overviews or AI Mode.

This article is for marketers, developers and site owners who want to know what needs to happen technically, and what doesn't. I cover AI crawlers, robots.txt decisions, JavaScript and speed, schema markup and llms.txt. How AI search engines then choose their sources, and what to do about content and reputation, is covered in my article on how AI search engines choose sources.

Which AI crawlers are there and what do they do?

AI crawlers fall into three groups: bots that collect content to train models, bots that build a search index, and agents that fetch a page live because a user is asking about it right now. That distinction is the basis for every robots.txt decision.

Crawler or tokenOwnerPurposeEffect of blocking
GPTBotOpenAIModel trainingSignals that your content shouldn't be used for training
OAI-SearchBotOpenAIShowing websites in ChatGPT's search featuresYour pages won't be shown as sources; at most as a navigational link
ChatGPT-UserOpenAIFetching a page when a user asks ChatGPT somethingNot an automatic crawler; according to OpenAI, robots.txt may not apply
OAI-AdsBotOpenAIChecking ad landing pagesOnly relevant if you advertise
PerplexityBot / Perplexity-UserPerplexitySearch index / live requestsLess or no visibility in Perplexity answers
ClaudeBot / Claude-SearchBot / Claude-UserAnthropicTraining / search / live requestsEach can be controlled separately
Google-ExtendedGoogleToken: use for training and grounding of Gemini models and appsNo effect on Google Search or AI Overviews
Applebot-ExtendedAppleTraining Apple's modelsNo training; Applebot itself keeps crawling
BingbotMicrosoftThe Bing index, which also underpins Microsoft CopilotGone from Bing and Copilot

Sources: OpenAI's documentation on its crawlers and the documentation of the other providers. Names and roles change now and then, so check them once a year.

Should you block AI bots in robots.txt?

Only block what you deliberately don't want, and separate training from search. If you want to be visible in AI answers, allow the search bots. Whether you allow training is a business decision, not an SEO requirement.

A common mistake is a blanket ‘block all AI’ rule. That also removes you from the search features of ChatGPT, Perplexity or Claude, when you may only have wanted to prevent training. A balanced robots.txt might look like this:

  • User-agent: GPTBot + Disallow: / (no training by OpenAI)
  • User-agent: OAI-SearchBot + Allow: / (do appear in ChatGPT search)
  • User-agent: ClaudeBot + Disallow: / and User-agent: Claude-SearchBot + Allow: /
  • User-agent: PerplexityBot + Allow: /
  • User-agent: Google-Extended + Disallow: / if you don't want Gemini to use your content

This is an example, not default advice. A publisher with paywalled content will make different choices from a service business that wants to be recommended.

Two caveats. Robots.txt is a request, not security: well-behaved bots respect it, but it isn't a lock on the door. And besides robots.txt, check your CDN, firewall or hosting plan. Some services have settings that block bots before they even read your robots.txt. You won't see that in your CMS, but you will in your server logs.

What exactly does Google-Extended do?

Google-Extended isn't a separate crawler but a token in robots.txt. It controls whether Google may use your content for training and grounding Gemini models and apps. It has no effect on Google Search or AI Overviews.

That's an important detail. Blocking Google-Extended in the hope of staying out of AI Overviews won't work. According to Google's documentation on AI features and your website, you control how you appear in AI Overviews and AI Mode with the same tools as regular snippets: nosnippet, data-nosnippet, max-snippet and noindex. Those also affect your regular search results. There's no switch that turns off AI features alone.

Why should content be in the HTML, not only in JavaScript?

Because you can't assume every AI crawler executes JavaScript. Google renders JavaScript, but most AI companies say very little about how their bots handle it. The safe assumption: anything that isn't in the HTML source doesn't exist for some of those bots.

Think of product information that only appears after a script has loaded, FAQ answers pulled into an accordion, or an entire single-page app that shows a blank page without JavaScript. Google's guide to generative AI features (May 2026) explicitly lists JavaScript SEO basics among the technical requirements, alongside crawlability, indexability, allowing snippets, good page experience and less duplicate content. Server-side rendering or static HTML solves this.

Speed is part of the same story. A slow server or heavy page wastes crawl capacity and gives users a poor experience. Keep your Core Web Vitals within the ‘good’ thresholds: Largest Contentful Paint (LCP) up to 2.5 seconds, Interaction to Next Paint (INP) up to 200 milliseconds and Cumulative Layout Shift (CLS) up to 0.1.

In October 2026 I migrated seomotion.nl and racemotion.nl from WordPress to static HTML on Netlify, with zero JavaScript. All the text sits directly in the HTML, so every bot reads exactly what a visitor sees. On PageSpeed Insights, racemotion.nl went from 71 to 100 on mobile, and LCP dropped from 6.2 to 1.4 seconds. Seomotion.nl scores 100 on mobile and desktop. For AI crawlers, that's the simplest scenario possible: nothing to render, nothing to guess.

Does schema markup help you appear in AI answers?

Not directly. Google says structured data isn't required for generative AI features and that there's no special schema.org markup for them. Google does recommend structured data for rich results, and it helps systems understand who you are and what a page is about.

Structured data (also called schema markup) is JSON-LD code that describes a page's content using a shared vocabulary: this is an organisation, this is the author, this is a product at this price. It isn't a secret ranking factor and it doesn't make up for weak content. The markup must always match what's visible on the page.

TypeUsed forWhat it gives you
Organization + sameAsYour business, logo, contact details and profiles elsewhereClarity about your brand as an entity
PersonThe author or specialist behind the contentLinks content to a recognisable expert
Article / BlogPostingArticles with author and dateContext on publication and freshness
Product + OfferProduct pages with price and availabilityRich results; together with Merchant Center, important for products in AI answers
LocalBusinessLocal branchesComplements your Google Business Profile
BreadcrumbListSite structureBreadcrumbs in search results, clear hierarchy

Since 2023, FAQPage markup only produces rich results on Google for authoritative government and health websites. You can still use it, but don't expect expandable questions in Google from it.

What is llms.txt, and is it worth adding?

llms.txt is a Markdown file in the root of your website with a short summary and links to your most important pages, intended for language models. Google Search doesn't use it, and there's no evidence that the major AI search engines read it systematically.

The proposal comes from Jeremy Howard (Answer.AI), dates from September 2024 and is published at llmstxt.org. Google's May 2026 guide is clear: llms.txt, other AI text files and special markup or Markdown versions aren't needed, because ‘Google Search ignores these files’. Google does call them harmless and potentially useful for other services.

So when does it make sense? Mainly for technical documentation, APIs and tools, where developers and AI agents need to find the right pages quickly. For a typical business website, it's a small job with no demonstrable effect on visibility.

Seomotion.nl has an llms.txt itself. Not because I expect it to get me into AI Overviews (Google says otherwise), but because it takes ten minutes, breaks nothing and gives agents a tidy entry point. I'd never prioritise it over a good XML sitemap or clean HTML.

How do you check your site for AI search engines?

A few targeted checks will tell you within an hour whether AI crawlers can read your site. This is the order I follow in an SEO audit.

  1. Read your robots.txt and look for rules for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended and any general User-agent: * with Disallow.
  2. Check your CDN, firewall and hosting for settings that block bots or challenge them with a captcha.
  3. View the HTML source (not the inspected DOM) of your key pages: are the main text, prices and answers in there directly?
  4. Use URL Inspection in Search Console to confirm Google can index the page and see what the rendered version shows.
  5. Check Bing Webmaster Tools: being indexed in Bing is the foundation for Copilot.
  6. Validate your structured data with the Rich Results Test or the Schema Markup Validator.
  7. Look at your server logs to see which AI bots visit, which pages they request and which status codes they get.

A short list to tick off:

  • Search bots from AI platforms deliberately allowed; training bots a conscious choice
  • Key content in the HTML, not only in JavaScript
  • No accidental noindex, nosnippet or overly strict max-snippet
  • Core Web Vitals within ‘good’
  • XML sitemap submitted in Search Console and Bing Webmaster Tools
  • Organization, Person and Article markup that matches the visible page
  • Canonicals and redirects in order, little duplicate content

Moving to a new platform? Build this into your plan; my article on website migration SEO explains how I approached it myself.

My advice: what comes first?

Start with access and readability: robots.txt, CDN and HTML that doesn't depend on JavaScript. That decides whether you're in the game at all. Then come speed and clean structured data. llms.txt sits at the bottom of the list: fine to have, but not a priority.

If you want to understand what happens under the bonnet, read how search engines work. If you'd like to learn this yourself, I cover it in my technical SEO training. And if you'd like me to take a look, see my approach to technical SEO. I don't guarantee citations, but I can make sure nothing technical stands in the way.

Sources

  1. Overview of OpenAI crawlers (OpenAI)
  2. AI features and your website (Google Search Central)
  3. Google's Guide to Optimizing for Generative AI Features on Google Search (Google Search Central, May 2026)
  4. The /llms.txt file (Jeremy Howard, Answer.AI, September 2024)
  5. Web Vitals (web.dev, Google)

Frequently asked questions

How can I tell whether GPTBot or other AI bots visit my site?

Check your server logs or the logging in your hosting or CDN and filter on user agents such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot. Analytics tools usually don't see bots, because bots don't execute JavaScript. Bear in mind that anyone can fake a user agent; the major providers publish IP ranges so you can verify genuine bots.

Does blocking GPTBot affect my Google rankings?

No. GPTBot belongs to OpenAI and has nothing to do with Google Search. According to Google, blocking Google-Extended doesn't affect Search or AI Overviews either. Your Google rankings depend on Googlebot.

Should I offer a Markdown version of my pages for AI?

Not for Google: Google says special Markdown versions and AI text files aren't needed and that Search ignores them. Clean, semantic HTML with clear headings is easy for every crawler to read. A separate Markdown version mainly adds maintenance.

What's the difference between llms.txt and robots.txt?

Robots.txt is a widely used standard that tells crawlers what they may and may not fetch. llms.txt is a 2024 proposal that gives language models a summary and a list of key links. Robots.txt controls access; llms.txt is at best a signpost, and Google Search ignores it.

Can structured data cause harm?

Yes, if the markup doesn't match what's on the page, for example reviews or prices visitors can't see. That can lead to a manual action and loss of rich results. Markup that honestly describes the visible content is safe.

Curious what SEO and GEO could do for your business?

Tell me where your business stands today and where you want to take it. In a short, no-obligation call we will work out which steps will make the biggest difference.

Get in touch →