Crawl & indexing
ai.robots.txt — Community Crawler Catalog
Look up community-maintained AI crawler names and references, alongside generated blocking configurations for several web-server environments.
Provider: ai.robots.txt community. Editorial listing by Gateway; not an owner claim.
Visit official destination ↗- Best suited to
- Investigating bot traffic and checking crawler labels before choosing access rules
- Owner next step
- Use the catalog as a starting point, verify the operator's current documentation and identify the crawlers you want to reach your site. Review entries individually before using any blocking configuration.
- Access
- Public repository; technical review and site permissions for any configuration change
- Cost
- Free. Public MIT-licensed project. Its generated files are designed to block listed crawlers and can restrict desired discovery if copied wholesale. User-agent matches are not verified identity, a security certification or a guarantee of compliance.
- Interfaces
- Website
- Source evidence
- Official public source · checked 2026-10-20. This is a description review, not an execution or compatibility test.
- Traffic and availability
- Not monitored. Bot visits, citations, uptime and successful delivery have not been measured by this directory.
Listing content is informational. Inclusion does not authorize installation, changes to your site, or payment. Check current provider terms and access policies.