A robots.txt file is a standard text file stored at your website's root directory (e.g. example.com/robots.txt). It instructs web search engine crawlers (such as Googlebot, Bingbot, and Applebot) which pages or directories they are allowed or disallowed to crawl, preserving your crawl budget and hiding private administrative endpoints.

According to the official RFC 9309 specification, when multiple rules match a specific URL path, the validator evaluates the longest matching pattern length. If an Allow and a Disallow rule have the exact same matching length, the Allow rule takes precedence. If no rules match the user-agent or wildcard (*), the path is allowed by default.

To block specific AI model scrapers from crawling your website content for training, add dedicated User-agent directive blocks with Disallow: / rules. For example: User-agent: GPTBot Disallow: / and User-agent: ClaudeBot Disallow: /. Our tool automatically audits and highlights AI scraper access permissions in real time.

Googlebot is Google's core web crawler used for indexing web pages for Google Search. Google-Extended is a dedicated standalone user-agent token that allows web publishers to control whether their site content can be used to train Google AI products (such as Gemini and Vertex AI) without impacting their organic ranking in Google Search.

Robots.txt specifies crawl permissions and restrictions (telling bots where NOT to go), whereas an XML Sitemap serves as a structured catalog of all public, canonical URLs on your site that you explicitly WANT search engines to discover, index, and surface in search results.

Not necessarily. While Disallow prevents Googlebot from downloading and crawling the page's HTML content, Google may still index the URL without content if other external websites link to it. To guarantee complete removal from search engine indexes, use a 'noindex' meta robots tag or HTTP response header instead.

A Sitemap Index is an XML file that lists multiple child sitemaps. The Sitemap standard limits any single sitemap file to 50,000 URLs or 50MB uncompressed. If your website has more than 50,000 pages, you must segment your URLs across multiple sitemaps and reference them inside a parent <sitemapindex> file.

The Crawl-delay directive requests bots to pause a specified number of seconds between consecutive page requests to reduce server load. Bingbot and Yandex respect Crawl-delay, but Googlebot does not support the directive in robots.txt (Googlebot crawl rate can only be configured in Google Search Console).

Major search engines like Google and Bing cache robots.txt files and re-fetch them multiple times a day (typically every 24 hours). XML sitemaps are parsed when search engines crawl your root domain or when submitted directly via Google Search Console and Bing Webmaster Tools.

Press '/' to instantly focus the input field, 'Ctrl+Enter' to trigger validation, 'Alt+V' to switch between 2-Column Split Studio and Stacked View, 'Alt+D' to bookmark this utility to your Starred tools (persisted in localStorage), 'Alt+P' for DevToys Smart Clipboard Auto-Detect, 'Alt+R' to cycle through realistic demo presets, 'Alt+S' to open the Developer API Code Studio modal with 1-click standalone script downloads across 6 languages (.ts, .py, .go, .rs, .php, .sh), 'Alt+N' to copy the live target export filename, 'Alt+M' for Markdown summary reports, 'Alt+J' for JSON audit payloads, and 'Alt+1' through 'Alt+6' for 1-Click Quick Copy across Disallowed paths, Allowed paths, Sitemap URLs, Clean Directives, Markdown tables, and JSON audit proofs.