Live robots.txt fetch and path rule simulation
Fetch a domain robots.txt file server-side, inspect sitemap declarations and crawler groups, test several paths for a crawler token, and see the selected group and winning allow or disallow rule instead of guessing from a raw text file.
The path simulator follows Google-style group selection and path precedence for common robots rules. Always verify critical crawl decisions with the crawler tools and live host behavior that matter to your site.
What a robots.txt tester should make clear
Robots.txt is easy to read quickly and easy to misread under pressure. A file may contain a general User-agent: * group, a more specific crawler group, an allow exception nested inside a broader disallow pattern, one or more sitemap declarations and comments that hide the important part at a glance. A useful robots.txt tester should show both views: the raw file you can audit line by line and the practical decision for the path you care about.
This tool fetches the live file at the domain root through the backend, parses crawler groups, tests several paths at once and explains which rule won for the selected crawler token. That makes it useful after WordPress changes, theme or plugin changes that alter virtual robots output, sitemap plugin changes, migration work and any Search Console report that mentions a blocked URL.
Robots.txt controls crawling, not everything about indexing
A robots rule tells compliant crawlers whether they may request a path. It is not a privacy wall and it is not a clean page-removal switch. If a URL is blocked from crawling, search engines may be unable to fetch page-level canonical or robots meta signals from that page. For index cleanup, choose the signal that matches the goal: redirects for moved content, noindex on fetchable content you do not want indexed, status codes for removed content, and robots rules for crawl access.
How crawler and path matching are read here
The simulator is designed around the decision a technical SEO needs to understand. It selects the most specific crawler group that matches the token you typed, merges equally specific matching groups, then evaluates allow and disallow path rules. The longest matching rule wins; when equally specific rules conflict, the allow outcome is preferred. It also supports the common * wildcard and $ end marker used in modern search-engine robots parsing.
- Use the exact host you want to audit. Robots rules are tied to the fetched robots.txt host and scheme context.
- Test a real public path, an admin or search path, and any URL reported as blocked.
- Read the winning rule, not just the rule count.
- Review declared sitemap URLs and confirm they still parse.
- Keep raw output visible when you need to compare the live file with plugin settings.
WordPress robots checks worth doing
For a public WordPress site, it is common to see a block for /wp-admin/ with an allow exception for /wp-admin/admin-ajax.php. That does not prove every crawl choice is right. You should still test valuable article and tool paths, search or parameter patterns you intentionally limit, sitemap discovery, and any rule injected by security, SEO or hosting layers.
Good technical SEO habits around robots.txt
- Fetch the live file after changing SEO plugins, sitemaps, caches or migrations.
- Test the same URL for Googlebot and the generic star group when rules look split.
- Do not block resources crawlers need to render public pages without a strong reason.
- Keep sitemap lines absolute and current.
- Pair robots checks with indexability, canonical and sitemap checks on pages that should rank.
Common questions
Does an allowed robots result guarantee indexing?
No. It only removes one crawl-access doubt. The URL still needs useful content, a healthy status, sensible canonical behavior, internal discovery and no conflicting noindex signal.
Is a blocked path always a mistake?
No. Admin, cart, search, duplicate or private workflow paths may be intentionally blocked. The question is whether the blocked path matches your site goal.
Why test several paths at once?
A robots file often has broad rules with narrow exceptions. Testing a public page, a blocked area and an exception path side by side makes the pattern easier to validate.
Does robots.txt stop a page from being indexed?
No. It blocks crawling, but a disallowed URL can still be indexed without a snippet if other pages link to it. To deindex, allow crawling and serve a noindex directive.
What is the difference between Disallow and noindex?
Disallow in robots.txt blocks crawling; noindex (a meta tag or header) tells search engines not to index. Use the right one: blocking a page you want deindexed actually prevents the noindex from being seen.
Where must robots.txt live?
At the root of the host, exactly at /robots.txt. A robots file in a subfolder is ignored. Each subdomain needs its own.
