Crawl & indexing
Known Agents — AI Bot Robots.txt Checker
Review which catalogued AI agents and search crawlers a public site's robots.txt allows or blocks, with explanations of the different bot categories.
Provider: Known Agents. Editorial listing by Gateway; not an owner claim.
Visit official destination ↗- Best suited to
- Finding accidental access restrictions affecting desirable AI and search crawlers
- Owner next step
- Check a public site you control and review the relevant search and user-directed agents individually. Decide on rules before separately deploying a change.
- Access
- Public checker accepting a website URL
- Cost
- Free. Listed as a free public tool. Provider analytics and automatic rule management are separate services. Robots rules are not authentication or a guarantee of compliance, citations or traffic.
- Interfaces
- Website
- Source evidence
- Official public source · checked 2026-10-20. This is a description review, not an execution or compatibility test.
- Traffic and availability
- Not monitored. Bot visits, citations, uptime and successful delivery have not been measured by this directory.
Listing content is informational. Inclusion does not authorize installation, changes to your site, or payment. Check current provider terms and access policies.