Websites and AI search: how ChatGPT, Gemini and AI Overviews find your site

How AI answers pick their sources

AI search tools don’t read the whole web from scratch each time you ask a question. They lean on search indexes and their own crawlers, then pick pages to quote or link. For your website, that means two things matter: the right bots can reach your pages, and your pages clearly answer the question.

Here is how the big ones work, according to their own documentation.

• Google AI Overviews and AI Mode use Google Search. Google’s guide to AI features and your website says a page must be indexed and eligible to show in Search with a snippet to appear as a supporting link. Both features may run several related searches across subtopics and data sources, which Google calls query fan-out.
• ChatGPT search can show any public website that doesn’t block OpenAI’s search crawler, OAI-SearchBot, according to OpenAI’s publisher FAQ.
• Claude uses Claude-SearchBot to index pages for its search answers. Anthropic says blocking it may reduce your visibility there.
• Perplexity uses PerplexityBot to surface and link websites in its results.
• Microsoft Copilot grounds its web answers by sending search queries to Bing, per Microsoft’s documentation. If Bing can’t find your site, Copilot likely can’t either.

Google’s verdict: it’s still SEO

Google’s 2026 guide to optimizing for generative AI features says this work is still SEO. Labels like GEO and AEO describe the same job. Google says there are no extra requirements or special optimizations needed for AI Overviews or AI Mode.

The same guide clears up a few popular tips. You don’t need to break content into tiny chunks for AI, and there’s no ideal page length. Structured data isn’t required for AI features, though it still helps with rich results. And mass-producing pages for every variation of a question can break Google’s spam policy on scaled content abuse, which applies to AI answers too.

So the website basics from our page on websites and SEO are also your AI search basics: clear titles, readable URLs, useful pages, and a site Google can crawl.

How many people use AI search

These are the companies’ own numbers, not independent measurements.

• Google said in May 2026 that AI Overviews has over 2.5 billion monthly active users, and AI Mode has passed 1 billion.
• Google said the Gemini app passed 900 million monthly active users, up from 400 million a year earlier.
• OpenAI said in March 2026 that ChatGPT has more than 900 million weekly active users and over 50 million subscribers.

Weekly and monthly users aren’t measured the same way, so don’t compare these figures directly. The point is simpler: hundreds of millions of people now get answers this way, and you want your site to be one of the sources.

The AI crawlers, and which ones you want

A crawler (or bot) is a program that visits web pages automatically. AI companies run three kinds, and they do very different jobs:

• Search bots index your pages so the AI tool can cite and link to you. Block these and you drop out of that tool’s answers.
• Training bots collect content to train future AI models. OpenAI, Google and Apple each say their training controls are separate from search, so blocking these doesn’t remove you from search answers.
• User fetchers open a page when a person asks the AI to look at it. Some don’t follow robots.txt, because a person asked for the page.

Sources: OpenAI’s crawler overview, Anthropic’s help article on its crawlers, Perplexity’s crawler docs and Apple’s support page on Applebot. OpenAI also lists OAI-AdsBot, which only visits pages submitted as ChatGPT ads.

To block one bot, your robots.txt file needs two lines: User-agent followed by the token (for example, GPTBot), then Disallow: / to cover the whole site. Anthropic says its bots honor robots.txt. Keep in mind that robots.txt is a request, not a lock. Wix’s help center puts it well: some crawlers may ignore these rules.

Microsoft works differently: Copilot answers come from Bing search, so Copilot isn’t in the table. Bing said in 2023 that pages with a NOARCHIVE robots meta tag are left out of its AI chat answers (now Copilot) and not used for training, while NOCACHE limits use to the URL, title and snippet.

Google’s three controls, and what each one does

Google gives you three different switches. People mix them up all the time.

1. Google-Extended is a robots.txt token, not a crawler. It controls whether content Google crawls is used to train future Gemini models and for grounding in some Gemini and Vertex AI products. Google says it doesn’t affect inclusion or ranking in Google Search, so it does not keep you out of AI Overviews.
2. The Search generative AI setting in Search Console (under Settings) excludes your site from AI Overviews, AI Mode and generative AI features in Discover. It reached all websites worldwide as of August 31, 2026. Google’s help page says sites that opt out get no traffic or impressions from those features, that the setting isn’t used as a ranking signal for the rest of Search, and that changes usually apply within a few days. It doesn’t affect AI training; that’s Google-Extended.
3. Page-level robots rules work page by page. A nosnippet rule stops content being used as a direct input for AI Overviews and AI Mode, but it applies to all Google result types, so your normal search snippet disappears too. Max-snippet limits how much text can be used, data-nosnippet hides parts of a page, and noindex removes the page from Search entirely. Google says these can take several days to several months to apply after a recrawl.

For most small businesses, the right setting is to leave all three alone. Opting out of AI features means giving up the traffic they send.

How to see if AI search sends you visitors

• Google: Search Console’s generative AI performance reports show how often your pages appeared in AI Overviews, AI Mode and Discover AI features, by page, country, device and date. All sites got them as of August 31, 2026. Clicks from AI features are also counted in the regular Performance report under Web.
• ChatGPT: OpenAI adds utm_source=chatgpt.com to links it sends, so the visits show up as their own source in your analytics.
• Copilot: Bing Webmaster Tools has an AI Performance report, in public preview since February 2026, that shows citations in Copilot and Bing AI summaries, which pages were cited and the queries behind them. Bing recommends clear structure, cited evidence, fresh content and IndexNow.

llms.txt: what it is and what it does

llms.txt is a proposal from Jeremy Howard of Answer.AI, published September 3, 2024. The idea is a Markdown file at the root of your site that gives AI tools a short summary and links to your most useful pages. llmstxt.org now presents an updated version from August 2026 that adds Markdown copies of pages at .md addresses. It is still a proposal, not a standard.

Google is clear about it. Its documentation says llms.txt and similar AI text files aren’t needed for Google Search, are ignored by it, and won’t help or hurt your visibility or rankings. Google’s AI features guide also says you don’t need new machine-readable files or special markup to appear in its AI features.

Some builders make the file for you anyway. Wix generates and maintains an editable llms.txt (on an upgraded site with a custom domain and indexing on). Hostinger’s AI Builder has an Enable LLMs.txt toggle. Webflow lets you upload one, Framer hosts one on Pro and Enterprise plans, and the Yoast SEO plugin can manage one on WordPress. Framer also serves a Markdown version of your pages to AI agents automatically.

Our view: if your builder makes llms.txt for free, leave it on. Don’t pay for it or spend hours on it. The crawler docs from OpenAI, Anthropic and Perplexity all describe robots.txt as the way to control their bots.

Check Cloudflare so you aren’t invisible

Cloudflare is a network service that sits between visitors and many websites, acting as a CDN and security layer. It says it handles traffic for about 20% of the web. If you or your web person ever pointed your domain’s DNS to Cloudflare, its settings apply to your site.

On July 1, 2025, Cloudflare began blocking AI crawlers by default and asking every new domain owner whether to allow them. It framed the change as blocking AI crawlers unless they pay creators. It also offers a managed robots.txt that asks training crawlers not to use your content, on all plans including Free.

The tool to check is AI Crawl Control (formerly AI Audit), which Cloudflare’s docs say is on all plans. It shows which AI services visit your site and lets you allow or block each crawler one by one. A pay per crawl option, which charges AI crawlers for access, is still in private beta.

If you never set up Cloudflare yourself, ask your builder how it handles AI bots. Webflow, for example, said the 2025 change caused no immediate changes for Webflow sites.

What your builder lets you switch on or off

• Squarespace: a Block known artificial intelligence crawlers checkbox under Settings, then Crawlers. It adds robots.txt rules for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others. It can’t target single pages and doesn’t remove content already collected. Squarespace’s help article covers it, and its AI search guide tells owners who want AI visibility to leave the box unchecked. Squarespace also offers an AI Visibility tool for English sites on version 7.1.
• WordPress.com: a Prevent third-party sharing checkbox under Settings, then Reading, for public sites. It keeps your content out of WordPress.com’s partner network and adds known AI bots to your robots.txt disallow list.
• Framer: its automatic robots.txt allows all AI and search crawlers by default. You can set noindex, restrict specific agents in site settings, or upload your own robots.txt on Pro and Enterprise.
• Wix: no one-click toggle. You add Disallow rules for specific bots in the Robots.txt Editor, or tick nosnippet on a page to keep it out of AI Overviews.
• Webflow: no dedicated toggle. You write custom robots.txt rules on a paid plan and can add content-signal preferences.
• Shopify: manages bot traffic at the network level and lets you allow or block specific AI crawlers through a robots.txt.liquid file. Shopify notes that blocking crawlers doesn’t stop its Catalog from sending product data to AI shopping agents you’ve turned on.
• Self-hosted WordPress: you control robots.txt directly. We build on WordPress every week and edit it through an SEO plugin.
• Hostinger, GoDaddy and Square Online: the help pages we checked don’t describe an AI crawler setting. Hostinger documents its llms.txt toggle only.

Not sure what your site says? Open yourdomain.com/robots.txt in a browser. Any line that pairs OAI-SearchBot, Claude-SearchBot or PerplexityBot with Disallow: / is keeping you out of those tools.

A short AI search checklist

1. Make sure Google can index your site, and connect Search Console.
2. Decide on training separately from search. Allow the search bots either way.
3. Check your builder’s AI setting (Squarespace’s crawler checkbox, WordPress.com’s third-party sharing box).
4. If your domain uses Cloudflare, review AI Crawl Control.
5. Read your robots.txt for Disallow lines aimed at search bots.
6. Add your site to Bing Webmaster Tools, since Copilot answers come from Bing.
7. Don’t turn on Google’s Search generative AI opt-out unless you have a clear reason to.
8. Skip the tricks: no tiny chunks, no pages for every phrasing of a question.

Everything above is about what your website allows. Getting cited more often is a marketing job. Our sister site, SEO PPC Management, covers AI search optimization.

What to do next

• Get the search basics right first with our guide to websites and SEO, which compares what each builder lets you control.
• Building a new site with AI tools? That’s a different topic: our AI website builders hub covers Lovable, Bolt, v0 and the builders’ own AI features.
• Want the full picture of search, ads, social media, email and ownership? Start at your website and everything around it.
• Thinking about switching platforms? Check how easy each one is to leave with our exit scores.