robots.txt Tester
Check whether a URL is crawlable, and see exactly which rule decided it.
Googlebot/proxies/italy/dashboard/settings/api/proxy/whitelist-ips/blogHow robots.txt rules are actually resolved
Most robots.txt confusion comes from three rules that are not obvious from reading a file top to bottom. This tester implements Google’s behaviour, so it will disagree with intuition in exactly the places Google does.
1. A crawler obeys one group, not all of them
Groups are selected by the most specific matching user-agent, and the winner is the only group that applies. If your file contains a Googlebot group and a User-agent: * group, Googlebot reads its own group and ignores the wildcard entirely — nothing is inherited. A Disallow: /admin/ sitting in the wildcard group does not restrict Googlebot at all.
User-agent tokens match case-insensitively as prefixes, so a Googlebot group also covers Googlebot-Image — unless a more specific Googlebot-Image group exists, which then wins for that crawler alone.
2. Longest match wins, not first match
Order in the file carries no meaning. Given Disallow: /admin/ and Allow: /admin/public/, the path /admin/public/logo.png is allowed, because the Allow pattern is longer — regardless of which line came first. Reordering the file changes nothing.
3. Ties go to Allow
When two matching rules have equal-length patterns, Allow wins. So Allow: /folder with Disallow: /folder permits /folder/page.
Disallow is not noindex
This is the costliest misunderstanding in the list. Disallow prevents crawling; it does not remove a URL from search results. A disallowed page can still be indexed — typically listed with no description — when Google discovers links to it.
Worse, the two directives conflict: if you block a URL in robots.txt and put a noindex tag on the page, Google can never crawl the page to see thenoindex, so the URL can stay indexed indefinitely. To deindex a page, allow crawling and serve noindex.
Testing what a crawler sees from another country
robots.txt is usually served identically worldwide, but the pages behind it often are not — and some sites serve different files, or block requests, by region. If you are auditing how a site behaves for crawlers in a specific market, you need requests that originate there. That is what our Italian and Spanish proxies are for, and the wider workflow is covered on proxies for SEO monitoring.
Privacy
Parsing and matching run entirely in your browser. The file you paste is never sent to our servers, and this page deliberately does not fetch robots.txt from a domain on your behalf — doing so would turn it into an open request proxy.
Frequently asked
Does a crawler combine the rules in "User-agent: *" with its own group?
No. A crawler obeys exactly one group — the one whose user-agent matches it most specifically. If your file has a Googlebot group and a wildcard group, Googlebot reads the Googlebot group and ignores the wildcard group completely. Rules you assumed were inherited from the wildcard group will not apply, which is the single most common robots.txt mistake.
If two rules match, does the first one in the file win?
No. Order in the file is irrelevant. The rule with the longest matching path pattern wins, whether it appears first or last. If two matching rules are the same length, Allow beats Disallow.
Does Disallow remove a page from Google?
No, and this trips people up constantly. Disallow stops crawling, not indexing. A blocked URL can still appear in results — usually with no description — if Google finds links to it elsewhere. To keep a page out of the index you need a noindex directive, and Google has to be able to crawl the page to see it. Blocking the URL in robots.txt actually prevents Google from ever reading your noindex.
What does the $ at the end of a pattern do?
It anchors the match to the end of the path. Disallow: /*.php blocks /index.php and /index.php?id=1, because patterns are prefix matches by default. Disallow: /*.php$ blocks only /index.php — the query-string version no longer matches.
Is robots.txt case sensitive?
The field names are not — user-agent and User-Agent are equivalent, and user-agent tokens are matched case-insensitively. The path patterns are case sensitive: Disallow: /Admin does not block /admin.
Why does this tool not fetch robots.txt from a domain?
Fetching arbitrary URLs from our server would make this an open request proxy, which is an abuse and SSRF risk we are not willing to introduce. Paste the file contents instead — you can view any site’s file at its /robots.txt. Everything here then runs in your browser, and nothing you paste is transmitted anywhere.
Need Proxies?
Dedicated and shared datacenter proxies across the USA, Spain, and Italy. Instant activation.