Gateway Information Group

Crawl & indexing

Robots Exclusion Protocol — RFC 9309

Read the formal robots exclusion rules for describing which URL paths cooperating crawlers may request.

Provider: IETF. Editorial listing by Gateway; not an owner claim.

Visit official destination ↗
Best suited to
Understanding robots rules without confusing them with security controls
Owner next step
Use the standard alongside each crawler’s guidance when reviewing public crawl rules. Protect confidential pages with authentication.
Access
Public specification; technical interpretation and site access for implementation
Cost
Free. Free public standard. Robots rules are not access authorization or a security boundary; compliance does not guarantee indexing, citations or referrals.
Interfaces
Website
Source evidence
Official public source · checked 2026-10-18. This is a description review, not an execution or compatibility test.
Traffic and availability
Not monitored. Bot visits, citations, uptime and successful delivery have not been measured by this directory.

Integration documentation ↗

A little direction

Put this resource to work

Get a practical first step based on this listing and its documented limits.

Optional · Free to use
Help me use this resource

Suggestions use our curated records and their access and cost limits. This does not scan your site or guarantee visits, citations or rankings. Privacy details.

Other listings in this category, ordered A–Z. Compare their purpose, access and cost before choosing.

Listing content is informational. Inclusion does not authorize installation, changes to your site, or payment. Check current provider terms and access policies.

Suggest a correction · Suggest your site