Crawl & indexing
Google robots.txt Guide
Understand how robots.txt controls crawling and why crawl restrictions differ from indexing and private-content protection.
Provider: Google. Editorial listing by Gateway; not an owner claim.
Visit official destination ↗- Best suited to
- Choosing intentional crawler access
- Owner next step
- Review accidental crawl blocks on useful public pages; use appropriate access controls for private content.
- Access
- Public documentation
- Cost
- Free. Free documentation; robots.txt is not an access-control mechanism.
- Interfaces
- Website
- Source evidence
- Official public source · checked 2026-10-01. This is a description review, not an execution or compatibility test.
- Traffic and availability
- Not monitored. Bot visits, citations, uptime and successful delivery have not been measured by this directory.
Listing content is informational. Inclusion does not authorize installation, changes to your site, or payment. Check current provider terms and access policies.