llms.txt and AI Crawlers: A robots.txt Guide for UAE Businesses

What robots.txt does, and what it does not

robots.txt is a plain text file at the root of your domain. It tells crawlers which paths they may fetch and which they should leave alone, grouped by the name of the crawler, which the file calls the user agent. A Disallow line under a user agent asks that crawler not to request the paths that follow. That is the whole mechanism.

Three limits matter for a business owner. First, it is a request. Well-run crawlers from Google, Microsoft, OpenAI, Anthropic and Perplexity honour it, but nothing in the file can stop a crawler that chooses not to. Second, it does not hide pages. A page blocked in robots.txt can still appear in search results if other sites link to it, just without a description, because the engine knows the address but was asked not to read the content. If you need a page kept out of results, use a noindex tag or password protection instead. Third, the file is public. Anyone can type your domain followed by /robots.txt and read your choices.

So robots.txt is the right tool for one job: telling well-behaved crawlers where not to spend their time. It is the wrong tool for privacy, for security, or for hiding a page you would rather nobody saw.

The AI crawlers by name, and who runs them

The confusion around blocking AI comes from the fact that each AI company now runs more than one crawler, and they do different jobs. One collects pages to train future models. Another fetches a page because a user asked a question and the assistant wants to cite a source. Another builds a search index that the assistant consults. Blocking all of them with one rule treats three different things as one.

OpenAI documents three. GPTBot gathers content that may be used to train models. OAI-SearchBot builds the index behind ChatGPT search, and blocking it removes your site from those results. ChatGPT-User fetches a page when someone asks ChatGPT about it directly. The OpenAI crawler documentation is explicit that blocking GPTBot does not remove a site from ChatGPT search; only blocking OAI-SearchBot does that.

Anthropic runs ClaudeBot for training data collection, Claude-SearchBot for its search index, and Claude-User for fetches a person triggers inside Claude. Anthropic's crawler documentation describes how each responds to robots.txt. Perplexity runs PerplexityBot to index pages for its answer engine, and Perplexity-User for pages a user asks about; Perplexity's documentation notes that user-initiated fetches may not follow robots.txt because a person requested the page.

Google is the one that catches people out. Googlebot indexes for Search and also supplies AI Overviews. Google-Extended is not a crawler at all but a control token: adding it to robots.txt tells Google not to use your content for training Gemini models or for grounding in Gemini apps. The Google crawler overview states that Google-Extended does not affect inclusion in Search or ranking. Bingbot feeds the Bing index that Microsoft Copilot draws on, so there is no separate Copilot crawler to manage. Applebot indexes for Siri and Spotlight, with Applebot-Extended as Apple's training control.

The real trade-off: training crawlers versus the crawlers that cite you

Once the names are separated, the decision becomes clearer. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) is a statement about whether your writing may be used to improve someone else's model. Reasonable people disagree, and it costs you nothing in visibility today. Blocking the search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) is a decision to disappear from the places where more and more of your prospects now ask their questions.

We see the mistake most often in a single block copied from a forum post: a rule that lists GPTBot, ChatGPT-User and OAI-SearchBot together, or one that names every crawler the author could find. The owner believes they have protected their content from training. What they have actually done is remove themselves from ChatGPT search, from Perplexity and sometimes from Bing, while leaving the training question largely untouched, because a model trained last year already read the site.

There is a second cost. Answer engines pick sources they can fetch and quote. If your competitor in the same Dubai industry allows the answer crawlers and you do not, the assistant will describe your market using their pages and their framing. How to be the source they pick is covered in our guide to generative engine optimisation and AI citations.

A sensible default for a company that wants to be found

For an office IT firm, a clinic, a law practice or a trading company in the UAE, the business case is almost always to be found. Here is the default we recommend and the one we use on our own site: allow Googlebot, Bingbot and Applebot without conditions; allow OAI-SearchBot, Claude-SearchBot and PerplexityBot so that answer engines can cite you; allow ChatGPT-User and Claude-User so that a prospect who pastes your address into an assistant gets an accurate reading of the page. Then make a separate, deliberate choice about GPTBot, ClaudeBot and Google-Extended. We have left them allowed, because our content exists to be read, and we would rather a model learn the correct version of what we do than a competitor's description of it.

Written as robots.txt rules, that default is short. The important part is that the only Disallow lines point at paths you genuinely do not want crawled, such as an admin area or an internal search results page, not at crawlers.

• User-agent: * followed by Disallow: /admin/ and Disallow: /search/ (or whichever paths you need to exclude for every crawler)
• No separate group for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot or Applebot, so they inherit the open default
• If you decide against training use: User-agent: GPTBot, User-agent: ClaudeBot and User-agent: Google-Extended, each followed by Disallow: /
• Sitemap: followed by the full address of your XML sitemap, on its own line

What llms.txt is, what it is not, and how to write one

llms.txt is a proposal, published at llmstxt.org, for a plain Markdown file at the root of your domain that tells AI assistants what your site is and where its most useful pages are. A model reading your site has limited attention and a lot of navigation, scripts and legal text to wade through. A short, curated map helps it reach the right page first. We publish one for exactly that reason.

It is worth being plain about what it is not. It is not a standard that Google, Microsoft or OpenAI require. No search engine has said that it affects ranking. It does not replace robots.txt, your sitemap or structured data, and it does not stop anyone from crawling anything. It is a courtesy document for machines, in the same way a well-written About page is a courtesy document for people. Treat it as cheap to do and modest in effect.

The format

The proposal is simple enough to write by hand. Start with a single top-level heading carrying your organisation's name. Follow it with one short blockquote that says what you do and for whom, in a sentence or two. Then add second-level headings for groups of pages (Services, Guides, Company), each followed by a list of links in Markdown form, with a short description after each link saying what the page answers. An optional final section can hold links that are useful but secondary, which an assistant may skip when it is short of room.

• Keep it to a page or two of text; it is a map, not a copy of the site
• Describe each link in the words a customer would use, not internal jargon
• Point at canonical URLs only, and update the file when pages move
• Serve it as plain text at /llms.txt, with the same care you give robots.txt

IndexNow: fast for Bing and Copilot, nothing for Google

IndexNow is a protocol that lets a site tell participating search engines the moment a page is added, changed or removed, instead of waiting to be crawled. You host a small key file, then send a single notification with the URL. Microsoft Bing, Yandex, Naver, Seznam and others share the notifications, so one ping reaches all of them. Because Copilot draws on the Bing index, IndexNow is the quickest route to having a new page available to Copilot answers.

The limit is that Google does not participate. A notification through IndexNow does nothing for Google Search or AI Overviews; Google still discovers pages through crawling, internal links and your XML sitemap, and you can request indexing of an individual page through Search Console. So IndexNow is worth doing because it is nearly free, but it is an addition to the Google-facing basics, not a substitute for them.

Verifying that it works: fetch it the way a crawler would

Every robots.txt change should be followed by a test, because the file fails silently. Nothing on your site looks different when you have accidentally blocked the whole thing. The simplest test is to fetch your own robots.txt in a browser and read it as a stranger would: which user agents are named, and what is each one asked to avoid? Then fetch a key page with a command-line tool, setting the user agent string to the crawler you care about, and confirm the page returns normally.

Google Search Console has a robots.txt report and a URL inspection tool that shows whether a page is crawlable and indexed; Bing Webmaster Tools offers the equivalent for the Bing index. For the AI crawlers, your web server logs are the honest source. Search them for the user agent names above and you will see whether the crawlers are visiting, how often, and whether they are receiving success responses. If a firewall or content delivery network sits in front of the site, check its bot rules too, because many ship with AI crawlers blocked by default regardless of what robots.txt says.

Common robots.txt mistakes we find on UAE business sites

Most of the damage we see is accidental. The file was written once by a developer, copied from a template, or left over from a staging environment, and nobody has read it since.

• Blocking CSS, JavaScript or image folders. Google renders pages like a browser, and a page it cannot style or script is judged as a broken page. Let crawlers reach everything a browser needs.
• Disallowing the sitemap or the folder it lives in, so the one file that lists every page is the one file crawlers may not read.
• A wildcard rule meant for one parameter that matches far more than intended, such as a pattern blocking every URL containing a question mark when your product or blog pages carry one.
• User-agent: * with Disallow: / left in from staging. The site is live, the owner is paying for advertising, and no search engine is allowed to read a page.
• Blocking the answer crawlers while meaning to block training, as described above.
• Case mismatches. Paths in robots.txt are case-sensitive, so a rule for /Admin/ does not cover /admin/.
• Relying on robots.txt to hide a page that should be behind a login or carry a noindex tag.

Where this sits in the wider work

robots.txt, llms.txt and IndexNow are plumbing. They decide whether crawlers can reach your pages and how quickly they hear about changes. They do not decide whether those pages are worth citing, which depends on the writing, the structure and the clarity of what you offer. Our SEO, AEO and GEO service covers both halves: we audit the crawl rules and publish the files, then work on the pages themselves so that search engines and assistants have something worth quoting.

If the site itself is the problem, with a platform that cannot serve a clean robots.txt or that injects blocking rules you cannot change, that is a build question rather than a content one. Our web development team builds sites where these files are part of the deployment, not an afterthought. And if you would like a second pair of eyes on your current robots.txt before changing anything, send it to us and we will tell you plainly what it allows and what it blocks.

Frequently asked questions

Written by Srinivasan Ramachandran, Founder, Listonics · Published Oct 7, 2026 · Last reviewed October 7, 2026 · 2,678 words