Robots.txt Checker
Enter a domain to fetch and analyze its robots.txt file.
About robots.txt
robots.txt is a plain-text file at the root of a website that tells crawlers (Googlebot, Bingbot, GPTBot, and others) which paths they may or may not request. The format is defined by the Robots Exclusion Protocol (RFC 9309) and is honoured by all major search engines and most well-behaved bots. This tool fetches the live robots.txt for any domain and parses each User-agent block.
The file is advisory, not enforcement: a malicious scraper can ignore it. But every legitimate search engine respects it, so it is the right place to keep crawlers out of admin paths, search-results pages, and infinite calendars. robots.txt also commonly references a Sitemap URL so crawlers can discover all canonical pages.
Frequently asked questions
Where should robots.txt be located?
At the very root of the domain, served at https://example.com/robots.txt with content type text/plain. Robots will not look anywhere else.
Does robots.txt block search engines?
It blocks crawling, not indexing. Google may still list a URL it has seen elsewhere even if it cannot crawl the page. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead.
What does User-agent: * mean?
A wildcard rule that applies to every crawler that does not have its own more specific block. Blocks for named user agents take precedence over the wildcard.
How do I let one bot in but block others?
Add a specific User-agent block (for example, User-agent: Googlebot with its own Allow rules) above the User-agent: * block. Each crawler obeys the most specific block matching its user agent.