Reads a robots.txt file and shows, for every LLM, whether its training, search and user crawlers are allowed or blocked, what that does to your AI visibility and where the file is costing you. Built from Should Publishers Block AI Training? Why Block It, When to Allow It and What the Data Says · How it works
Nothing you paste is retained or shared, not even for learning purposes. The file is read in your browser and goes nowhere else.
T train · S search and cite · U open on request, green allowed, red blocked. Blocked means the search crawler is blocked, so the LLM cannot cite you whatever the other letters say, and Cite only means training is blocked while search is open. A letter is left out where the LLM has no crawler for that job.
That is the short answer. The full audit sits in the tabs below: Start with Summary for the cost and the fixes, then Worth checking for the things to confirm. The rest is reference.
* These are suggestions: Confirm them against the business and its IP goals before changing anything, because every site is different. To test a change, edit the robots.txt and paste it here again, it does not need to be live first.
Each line ends with a call: Almost certainly not intended, worth confirming, or FYI.
By folder group
The paths the file treats alike, grouped, with each LLM's verdict and letters for that group. This is the view the tiles link to.
Path by path
One row per path that behaves differently from the site default, most restrictive first, and what each LLM's crawlers can do there. T train, S search and cite, U open on request. Green allowed, red blocked. A letter is left out where that LLM has no crawler for the job.
Googlebot and Bingbot first, then DeepSeek and Grok, then Common Crawl and the LLM-first crawlers most people have never heard of. Full list at knownagents.com/agents.
| Crawler | Type | Job | Status | What it means |
|---|
Scope: This audits the AI visibility side of robots.txt only. For syntax errors, Googlebot and Bingbot access, sitemap lines or anything else about search engine crawling, use Google Search Console's robots.txt report, Bing Webmaster Tools, TechnicalSEO.com's robots.txt tester or Screaming Frog.