{"id":2101,"date":"2026-10-25T03:26:32","date_gmt":"2026-10-25T03:26:32","guid":{"rendered":"https:\/\/seoscanpro.ai\/blog\/google-robots-txt-rules-ai-crawlers-encoding\/"},"modified":"2026-10-26T14:24:03","modified_gmt":"2026-10-26T14:24:03","slug":"google-robots-txt-rules-ai-crawlers-encoding","status":"publish","type":"post","link":"https:\/\/seoscanpro.ai\/blog\/google-robots-txt-rules-ai-crawlers-encoding\/","title":{"rendered":"Google Explains robots.txt Rules for AI Crawlers, Emojis, and International URLs"},"content":{"rendered":"

Two habits come from Google’s guidance on robots.txt: the file should follow the internet standard RFC 9309, and you should use percent-encoding for non-English characters in rule paths. Google’s Gary Illyes and Cherry Prommawin laid out both in the second video of their Non-Canonical Cabinet series, What can you actually put in a robots.txt?<\/em>, recapped by Search Engine Roundtable on October 21, 2026. The guidance separates what controls crawling from what governs AI training, and it shows where a rule can look right to a person and still do nothing.<\/p>\n

What Google said about robots.txt<\/h2>\n

Illyes and Prommawin described robots.txt as a plain-text file that lives at the root of a website and tells automated bots which paths they may and may not access. The rules apply to search engine crawlers, AI training crawlers, and other well-behaved crawlers, and a compliant crawler checks this file before it looks at any content on the site.<\/p>\n

Their top-level points, as presented in the video: Googlebot uses robots.txt to limit crawling, the file should follow the internet standard RFC 9309<\/a>, and you should use percent-encoding for non-English characters in rule paths. Google’s parser reads the file line by line and recognizes a small set of directives: user-agent, allow, disallow, and sitemap. Comments begin with a hash and are ignored, so a line of ASCII art in a comment is valid but only wastes bandwidth.<\/p>\n

The practical warning, which Search Engine Roundtable drew out of the same video, is that the parser silently discards any line it does not understand and keeps going. Write don’t crawl secret page<\/em> and Google reads don’t crawl<\/em>, sees no valid rule, drops the line, and may crawl the page you thought you had blocked.<\/p>\n

Which robots.txt files are affected?<\/h2>\n

This matters for any site whose robots.txt uses raw international characters or emojis in its paths, relies on non-standard syntax, or tries to set an AI training policy through a custom rule. Open your site’s \/robots.txt<\/code> at the root and read it the way the parser does.<\/p>\n

Check three things. Every rule uses one of the recognized directives rather than plain-English instructions. Any path with non-English characters or emojis should appear in its percent-encoded form rather than as raw multibyte characters, because Googlebot handles multibyte UTF-8 behind the scenes but other or older crawlers might get confused, and a crawler that cannot process the raw characters may ignore the whole disallow line.<\/p>\n

What to do now<\/h2>\n

Use this as an audit checklist<\/a> for the file, not a set of commands to paste in:<\/p>\n

    \n
  1. Confirm the file follows RFC 9309 and uses only the recognized directives: user-agent, allow, disallow, and sitemap. The parser silently discards any line it does not understand.<\/li>\n
  2. Test the actual URL patterns instead of trusting a rule because it reads clearly to a person. A robots.txt validator or Google’s own robots.txt documentation is the kind of tool for this; ask whoever maintains the site to run the patterns through one.<\/li>\n
  3. Percent-encode any non-English characters or emojis in rule paths, mirroring the form your server expects, so crawlers that do not process raw multibyte characters still read the rule.<\/li>\n
  4. Review the user-agent tokens in the file.<\/li>\n
  5. Keeping a specific AI crawler off your pages is up to whoever maintains the site.<\/li>\n<\/ol>\n

    Crawling control is not an AI training policy<\/h2>\n

    Illyes explained that, as defined by RFC 9309<\/a>, robots.txt is an access-preference protocol. It tells a bot that it may crawl a page or that it should not, and nothing more. It does not dictate what a system does with data after it has already crawled it. Blocking an AI crawler from reaching a page is a separate thing from telling a company not to train on content it already holds.<\/p>\n

    Perplexity’s crawler documentation states that PerplexityBot<\/a> "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models." In a separate case, Anthropic documents that it respects robots.txt for its ClaudeBot crawler, according to IEEE Spectrum<\/a>. The lesson from both: find the token the vendor actually publishes, because the file only acts on tokens it recognizes.<\/p>\n

    How many sites now block AI crawlers<\/h2>\n

    A study by researchers at Saarland University, IIT Bombay, and MPI-INF, published on arXiv<\/a>, found that 60.0% of reputable news sites disallow at least one AI crawler in robots.txt, compared with just 9.1% of misinformation sites, and that reputable sites forbid an average of 15.5 AI user agents while misinformation sites prohibit fewer than one. The same study reports that AI-blocking by reputable sites rose from 23% in September 2023 to nearly 60% by May 2025.<\/p>\n

    The pressure behind that shift is documented elsewhere. In an interview with IEEE Spectrum about the Data Provenance Initiative report "Consent in Crisis: The Rapid Decline of the AI Data Commons," lead researcher Shayne Longpre said the robot exclusion protocol was introduced in 1995 and that robots.txt is machine-readable but "does not appear to be legally enforceable," while terms of service may be enforceable but are not machine-readable. He added that writing robots.txt rules puts the burden on a site to identify, among thousands of crawlers, which ones it wants to allow or disallow.<\/p>\n

    That burden is heavy for the people most affected. A study of 182 professional artists<\/a> by researchers at UC San Diego and the University of Chicago found strong demand for tools like robots.txt, constrained by hurdles in technical awareness, agency in deploying them, and limited efficacy against crawlers that do not respond. The same paper cites an Akamai analysis suggesting roughly 70% of website traffic comes from automated crawlers against 30% from humans, and notes the concerns have produced over thirty ongoing copyright lawsuits.<\/p>\n

    Why robots.txt is not a security tool<\/h2>\n

    The file requests that compliant crawlers stay out of the paths you list. It does not prevent access by unauthorized clients and does not guarantee every bot will comply. Encoding inconsistencies can undermine a rule across different crawler implementations, which is why Google recommends the encoded form for anything beyond plain ASCII.<\/p>\n

    FAQ<\/h2>\n

    Can robots.txt stop AI companies from training on my content?<\/h3>\n

    No. Google explained that robots.txt is an access-preference protocol under RFC 9309. It tells bots which pages they may or may not crawl and does not control what a system does with data it has already obtained. To stop a specific AI crawler from accessing your pages entirely, you have to target its specific user-agent token explicitly in the file.<\/p>\n

    Can I use emojis or non-English characters in robots.txt?<\/h3>\n

    User-agent tokens accept only letters and dashes, so emojis and non-English characters do not work there. In rule paths, Google’s parser can handle multibyte UTF-8 characters, but older or other crawlers may not, so Google recommends the percent-encoded form to stay safe across all crawlers.<\/p>\n

    What directives does Google’s robots.txt parser recognize?<\/h3>\n

    The parser reads the file line by line and looks for user-agent, allow, disallow, and sitemap. It silently discards lines it does not understand, and comments that begin with a hash are ignored.<\/p>\n

    Try the site audit tool<\/h2>\n

    \"The<\/a><\/p>\n

    SEOScanPro runs a full technical audit of a site and shows the measured result behind every check. Open the site audit tool<\/a>.<\/p>\n

    \n

    Sources<\/h2>\n