ZPages All articles
SEO & Site Structure

A Bouncer with Last Decade's Guest List: Auditing robots.txt in 2026

ZPages
A Bouncer with Last Decade's Guest List: Auditing robots.txt in 2026

Photo: Caspiax, Public domain, via Wikimedia Commons

There is a file on your website that has probably not been read by a human since the day it was uploaded. It is four hundred bytes long, it sits at the root of your domain, and in 2026 it is making decisions on your behalf about search engines, AI assistants and whether your stylesheets are allowed to load.

It's called robots.txt. Let's go and look at it.

One file, nobody's job

Open a private browser window and type your domain followed by /robots.txt. Most UK business owners have never done this. What usually comes back is a short block of text, sometimes with a date in a comment revealing it was last touched before the current website launched.

Worth knowing what happened in between: the format was formally standardised as RFC 9309 in 2022, and the cast of crawlers reading it has changed substantially. The file hasn't.

What it actually does — and doesn't

robots.txt is a set of requests, not a lock. Well-behaved crawlers read it before fetching anything else. It has three jobs: allow, disallow, and point at your sitemap.

It cannot keep a page out of Google's index. That requires a noindex tag or a removal request. It cannot hide a page from a determined scraper, and it is not a security control — anything in it is public, including the paths you were trying to keep quiet.

Two behaviours matter more than people realise. Google fetches robots.txt before crawling anything on the host. If the file returns a server error, crawling can slow or pause. If it returns a 404, Google treats that as 'allow everything', which is usually harmless. If a firewall blocks it with a 403, or it redirects to your homepage, or it gets served as an HTML page, you have a broken file and no easy way to notice.

The staging copy that never left

The most common fault on UK SME sites is a leftover from launch: a single line, Disallow: /, copied across from the staging environment where it was doing a useful job. It blocks everything. The site looks fine to customers. Search Console quietly reports that pages are blocked by robots.txt and nobody reads that email.

Check it directly: put a live URL through URL Inspection in Search Console and confirm the page reports as allowed. Then check whether page resources are loading.

The rendering problem

The second most common fault is a disallow line that was once sensible. Blocking /wp-content/ or /assets/ or /includes/ seemed tidy to somebody. It also blocks your CSS, your JavaScript and much of your imagery.

Google renders pages in a browser-like environment. If the stylesheet can't be fetched, it may see a page that looks nothing like the one your customers see — and index accordingly. Images blocked at the folder level may not surface in Google Images either. This is not theory; it shows up as pages with thin, oddly structured results months after the fact.

Two other traps. Wildcards are supported — * for sequences, $ for end of URL — and a hastily written Disallow: /*? will cheerfully kill every URL with a parameter, including your own filters and tracking links. And robots.txt is per host: shop.yourdomain.co.uk or booking.yourdomain.co.uk needs its own file, or it has none at all.

Now the AI part

This is where the file has quietly become more interesting. Four groups of crawler are worth distinguishing:

Search crawlers. Googlebot and Bingbot build the index behind both classic results and AI Overviews, Copilot and other answer surfaces. Blocking these is almost always a mistake.

AI answer crawlers. OAI-SearchBot fetches pages for ChatGPT search. PerplexityBot does the same for Perplexity. Claude-SearchBot and Claude-User handle the equivalent at Anthropic. Blocking these means you simply don't appear when someone asks an assistant for a firm like yours.

AI training crawlers. GPTBot, ClaudeBot, CCBot (Common Crawl), Applebot-Extended, meta-externalagent. These collect text to train models. Blocking them won't hurt your rankings, but your content won't be in the training data either.

Google-Extended. The confusing one. It governs use of your content for Gemini training and grounding — not for Google Search or AI Overviews. People block it believing they've opted out of AI answers and have done precisely nothing of the sort. To appear in AI Overviews you need Googlebot allowed.

Whether you permit training crawlers is a business decision, not a technical one, and it deserves to be made deliberately rather than flicked on by a hosting toggle somebody enabled in 2025. Reach versus attribution. There's a respectable argument on both sides; the error is not choosing.

A ten-minute audit

  1. Load /robots.txt in a private window. Is it plain text? Does it load without redirecting?
  2. Read every Disallow line aloud. Can anyone explain why it's there?
  3. Confirm the launch-day Disallow: / never shipped to live.
  4. Check that CSS, JavaScript and image folders are crawlable.
  5. Confirm the Sitemap: line points at the current sitemap on the right host, and that the URL returns a 200.
  6. Run a live page through URL Inspection — allowed, and resources loading.
  7. Write down your position on AI training crawlers, with a date.
  8. Repeat for any subdomain running a shop, portal or booking system.

A short brief for your developer

Paste this, along with your decisions, and ask them to keep the file in version control with a date at the top.

# robots.txt for example.co.uk — reviewed 15 November 2026

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search
Disallow: /*?s=
Disallow: /thank-you/

# Search engines — keep these allowed
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI assistants fetching pages on a user's behalf
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# AI training crawlers — remove these blocks to allow training use
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.co.uk/sitemap.xml

Ask for the date comment to be updated every time the file changes. It's the cheapest documentation you'll ever get, and it means the next person to open the file has a fighting chance of knowing what they're looking at.

All Articles

Related Articles

Who Holds the Keys? A Website Ownership Audit for UK Businesses

Who Holds the Keys? A Website Ownership Audit for UK Businesses

Good Pages Don’t Need a Treasure Map: A Click-Depth Audit for UK Service Sites

Good Pages Don’t Need a Treasure Map: A Click-Depth Audit for UK Service Sites

The Phone Test: Does Your Navigation Survive Being Said Out Loud?

The Phone Test: Does Your Navigation Survive Being Said Out Loud?