More of your future clients will meet your business inside an AI answer before they ever see your homepage. Someone asks ChatGPT for "a WooCommerce developer who can fix a slow store", or Google shows an AI Overview at the top of the results, and a handful of websites get named. The rest don't exist in that moment.
Table of Contents
The good news: AI search doesn't need a new kind of website. It needs a website that machines can actually read. Most of the problems I find are old, boring technical issues that now cost more than they used to. This guide explains how AI tools read your site, then walks through what to check and fix, in order.
How AI search actually reads your website
There are two ways an AI tool learns about your site, and they work differently.
- Training data. Crawlers collect public pages to train future models. What's in a model's memory is months old and you can't update it quickly.
- Live search. When someone asks a question, tools like ChatGPT search, Perplexity and Google's AI Overviews look up current pages, read them, and write an answer with links. This is where you can win traffic now, and it's what this guide focuses on.
For live search, the process is roughly the same everywhere: a crawler fetches your page, pulls out the text, decides what the page is about and whether it can be trusted, then quotes or summarises the parts that answer the question. Every step can fail on a WordPress site for ordinary reasons.
Who is crawling you
Each AI company uses named crawlers. These are the ones worth knowing (see OpenAI's crawler documentation and Google's guide to AI features in Search for the official details):
| Company | Crawler | What it does |
|---|---|---|
| Googlebot | Normal Google Search crawling. AI Overviews and AI Mode are built on the same search index, so if Google can index you, you are eligible. | |
| Google-Extended | A robots.txt control for Gemini model use. Blocking it does not remove you from AI Overviews. | |
| OpenAI | OAI-SearchBot | Finds pages to show in ChatGPT search results. Allow this one if you want to be cited. |
| OpenAI | GPTBot | Collects pages for model training. You can block it without leaving ChatGPT search. |
| OpenAI | ChatGPT-User | Visits a page when a user asks ChatGPT to open it. |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers. |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Training, search indexing and user-requested visits for Claude. |
| Microsoft | Bingbot | Bing's index feeds Microsoft Copilot answers. |
The important split: search crawlers bring visibility, training crawlers don't. You can make a separate choice for each.
Fix 1: Make sure the crawlers aren't blocked
This is the most common problem I find, and the owner usually has no idea. Check these four places:
- robots.txt. Open
yoursite.com/robots.txt. Look for lines that block the search crawlers above, or a blanketDisallow: /. In Rank Math you can edit it under General Settings → Edit robots.txt. - Cloudflare. Cloudflare offers settings that block AI crawlers, and some of them are easy to switch on without noticing. Check the bot and AI crawler settings in your dashboard and allow the search bots you want.
- Security plugins and firewalls. Aggressive rate limits or "block unknown bots" rules in Wordfence or your host's firewall can turn crawlers away. Check the blocked traffic log for the crawler names.
- Noindex tags. A page set to noindex in your SEO plugin, or the "Discourage search engines" box under Settings → Reading, keeps you out of Google and therefore out of AI Overviews.
A sensible starting point if you want AI search visibility but don't want your content used for training:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Blocking training is a business choice, not a technical one. If you want the widest reach, allow everything.
Fix 2: Put your content in the HTML, not behind JavaScript
Googlebot runs JavaScript. Many AI crawlers don't: published analyses of their traffic show most of them read the raw HTML your server sends and stop there. If your important text only appears after scripts run, those crawlers see an empty page.
On WordPress, watch out for:
- Pricing tables, FAQs or service details loaded by a JavaScript widget.
- Tabs and accordions that fetch their content after a click.
- Product details on WooCommerce stores pulled in by an app or iframe.
- Page builders that wrap text in so many scripts that the real content is hard to find.
Quick test: open your page, right-click and choose "View page source" (not Inspect). Press Ctrl+F and search for a sentence from the page. If it's not in the source, assume AI crawlers can't read it.
This is one of the reasons I build themes by hand. The content is in the HTML from the first byte. I explain the wider trade-off in why hand-coded WordPress still wins over AI website builders.
Fix 3: Speed and reliability
A crawler that waits too long or gets an error moves on. Live AI search tools fetch pages while the user is waiting for an answer, so slow servers are a real disadvantage.
- Aim for a server response (TTFB) under about 600 ms. Page caching does most of the work.
- Avoid 5xx errors and timeouts during traffic spikes. Your host's error log will tell you.
- Remove plugins that load scripts on every page but are used on one.
If your site is slow, the real cost of a slow WordPress site shows what it's already costing you in leads, before AI is even considered.
Fix 4: Write pages that answer questions directly
AI tools quote passages that answer the question cleanly. Long introductions and vague marketing lines rarely get quoted. Changes that help:
- Answer first. Put a one- or two-sentence direct answer under each heading, then explain.
- Use question-style headings that match how people ask: "How much does a WooCommerce speed fix cost?" beats "Our Pricing Philosophy".
- Be specific. Prices, timeframes, locations and numbers are easier to quote than "affordable" or "fast turnaround".
- One topic per page. A page that tries to cover every service ends up quoted for none of them.
- Use real HTML structure: one H1, headings in order, lists and tables for lists and comparisons.
Fix 5: Tell machines who you are
AI tools try to judge whether a source is trustworthy. Make that easy:
- Schema markup. Add Organization or LocalBusiness, Person (for author pages), Article, Product and FAQ schema where they fit. Rank Math and similar plugins handle most of it. Schema doesn't guarantee a mention, but it removes guesswork about what the page is.
- Consistent details. The same business name, location and description on your site, Google Business Profile, LinkedIn and directories.
- Real author information. A named author with a short bio and experience beats "Admin".
- Clear About, Contact and pricing pages. These are often the pages an AI tool reads to describe your business.
Fix 6: Help them find new pages quickly
- Keep your XML sitemap clean and submitted in Google Search Console.
- Add your site to Bing Webmaster Tools. It takes ten minutes, and Bing's index feeds Copilot.
- Turn on IndexNow (Rank Math has it built in) so Bing and other supporting engines hear about new and updated pages straight away.
- Link new posts from older, related ones so crawlers reach them.
What about llms.txt?
llms.txt is a proposed file that lists your most important pages for AI tools, similar in spirit to a sitemap. It's cheap to add, but none of the major AI search providers has confirmed they use it. Do the fixes above first. Treat llms.txt as optional.
A 15-minute checklist
- Open
/robots.txtand confirm the search crawlers aren't blocked. - Check Cloudflare and your security plugin for AI or bot blocking.
- View the source of your three most important pages and find your key text in it.
- Test your homepage speed and fix anything over a few seconds.
- Rewrite the first paragraph under each main heading as a direct answer.
- Confirm schema is set up for your business and author.
- Submit your sitemap to Google Search Console and Bing Webmaster Tools.
- Ask ChatGPT, Perplexity and Google about your service in your city and see who gets named.
Related reading
- Why hand-coded WordPress still wins over AI website builders in 2026
- The real cost of a slow WordPress site: a lead-loss breakdown
- How to migrate off Elementor without breaking SEO
Want me to check how AI crawlers see your site?
Start with the free WordPress audit tool. It takes about 30 seconds and flags speed, SEO basics and outdated plugins. If you want a full check of robots.txt, firewall rules, rendering and schema, tell me about your site on the hire a freelance WordPress developer page. I reply with a short written plan and a fixed price.
Get the free WordPress Security Checklist 2026
25-point checklist PDF - malware detection, hardening guide, login security. Used by 500+ WordPress site owners.
- ✓ 25-point security checklist PDF
- ✓ WordPress malware scan guide
- ✓ Hardening checklist for any WordPress site
No spam. Unsubscribe any time.
You're in!
Check your inbox - the checklist PDF is on its way.
Need help with your WordPress site?
I'm a freelance WordPress developer who fixes exactly this kind of problem.
150+ projects. Clients in UK, US, UAE & Ireland. Fast turnaround.

Rajan Gupta
Freelance WordPress DeveloperI'm Rajan Gupta, a freelance WordPress developer based in India with 150+ projects delivered for agencies and businesses in the UK, US, UAE, and Ireland. I specialise in performance optimisation, Core Web Vitals, custom Sage/ACF builds, and WooCommerce development - every project ships with a 90+ PageSpeed baseline.