Gateway Information Group

AI & Bot Directory

Find useful sites, visibility tools and agent resources. Help AI find your corner of the web. Free to browse. Clear sources. No paid placement.

A little direction

Find your next step.

Describe your goal. Get a short path through the AI and bot resources in this directory.

Optional · Free to use
Help me choose resources

Suggestions use our curated records and their access and cost limits. This does not scan your site or guarantee visits, citations or rankings. Privacy details.

Explore the repository

Free discovery. Clear costs. No paid placement.
Providers may charge. Traffic and uptime are not monitored. Curated by Gateway from public sources; inclusion does not imply owner submission or endorsement.

Link to these results

Filter by a documented interface; inclusion is not a compatibility test.

91 listings to explore

A little direction

Choose a useful next step

Get suggestions that respect your current filters.

Optional · Free to use
Help me choose from these results

Uses your current search, category, interface and free-option filter. General resources are excluded from discovery suggestions.

Suggestions use our curated records and their access and cost limits. This does not scan your site or guarantee visits, citations or rankings. Privacy details.

A2A Protocol — Agent Discovery

Describe an agent's public capabilities with an Agent Card and review well-known URI, registry and direct-configuration discovery strategies.

Agent connectionsFree
Details

Describe an agent's public capabilities with an Agent Card and review well-known URI, registry and direct-configuration discovery strategies.

Owner next step If you operate an A2A service, publish accurate non-secret capability metadata and document authentication separately from public discovery.

Access Public documentation; an implemented A2A service is required to publish its own capabilities

Cost Free. Free public protocol guidance. Building, hosting and invoking an agent can have separate costs; an Agent Card does not authorize operations or guarantee traffic.

Interfaces Website · HTTP

Provider A2A project / Linux Foundation

Source evidence Official public source · Source checked 2026-10-18

  • Agent developers
  • A2A
  • Agent Cards
  • Capability discovery

Ahrefs — Brand Radar

Explore tracked AI brand mentions, cited pages and share-of-voice estimates, or configure selected prompts for monitoring supported AI platforms.

AI visibilityCheck terms
Details

Explore tracked AI brand mentions, cited pages and share-of-voice estimates, or configure selected prompts for monitoring supported AI platforms.

Owner next step Compare the index's topic coverage with your own audience, review included checks and paid limits, and measure referrals separately from modelled visibility.

Access Ahrefs account and the appropriate paid product or plan for the selected features

Cost Check current provider terms. Paid index and custom-prompt packages have different limits. Included checks require a qualifying paid plan; a preview is not an ongoing free monitoring plan.

Interfaces Website

Provider Ahrefs

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Brand monitoring
  • Prompt tracking

Ahrefs — Web Analytics

Measure website visits and AI referral sources with hosted analytics, alongside landing-page, campaign and outbound-link information.

AI visibilityFree plan
Details

Measure website visits and AI referral sources with hosted analytics, alongside landing-page, campaign and outbound-link information.

Owner next step Verify ownership, review privacy requirements and install the analytics snippet only on your own site. Compare referral visits with useful actions and separate crawler logs.

Access Ahrefs account, verified site ownership and permission to add its JavaScript snippet

Cost Free plan. Free for verified sites up to one million events per project each month at review time; higher limits require an add-on. Browser analytics is not a complete crawler or citation record.

Interfaces Website

Provider Ahrefs

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI referrals
  • Web analytics
  • Traffic measurement

Ahrefs Free

Explore the free owner toolkit, formerly Ahrefs Webmaster Tools, for website audit and search foundations.

AI visibilityFree plan
Details

Explore the free owner toolkit, formerly Ahrefs Webmaster Tools, for website audit and search foundations.

Owner next step Verify your site and review supported audit findings; check which features require a paid subscription.

Access Account and ownership verification for owner features

Cost Free plan. Free owner features have limits; paid subscriptions and AI visibility products are separate.

Interfaces Website

Provider Ahrefs

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

Amazon — Search and Crawler Guide

Distinguish Amazonbot, Amzn-SearchBot and Amzn-User, including search eligibility, training roles and official crawler verification information.

Crawl & indexingFree
Details

Distinguish Amazonbot, Amzn-SearchBot and Amzn-User, including search eligibility, training roles and official crawler verification information.

Owner next step Review each agent's purpose and published IP information. Choose search and training rules separately; Amazon notes user-initiated requests may not follow every robots.txt directive.

Access Public documentation

Cost Free. Free public documentation. Search eligibility is not a promise of Alexa placement, citations or referrals.

Interfaces Website

Provider Amazon

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Amazonbot
  • Alexa
  • Crawler controls

Anthropic Crawler Guide

Read official guidance on Anthropic crawling and the controls available to website owners.

Crawl & indexingFree
Details

Read official guidance on Anthropic crawling and the controls available to website owners.

Owner next step Read the current bot roles and robots.txt guidance before choosing access rules for your site.

Access Public help article

Cost Free. Free guidance; access settings are not a guarantee of discovery.

Interfaces Website

Provider Anthropic

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Crawler access
  • Documentation

APIs.guru

A directory of public REST API definitions with a route for contributing API descriptions.

API & server registriesCheck terms
Details

A directory of public REST API definitions with a route for contributing API descriptions.

Owner next step Review the contribution requirements for a public API definition.

Access Public directory

Cost Check current provider terms. Consult current contribution requirements and each listed API’s own access terms.

Interfaces Website · OpenAPI

Provider APIs.guru project

Source evidence Official public source · Source checked 2026-10-01

  • OpenAPI
  • API discovery
  • Documentation

APIs.json Discovery Format

Publish a machine-readable index linking your API definitions, documentation, pricing and service terms.

Agent connectionsFree
Details

Publish a machine-readable index linking your API definitions, documentation, pricing and service terms.

Owner next step Describe your own APIs with accurate public links; keep discovery metadata separate from authorization to execute operations.

Access Public format documentation

Cost Free. Open public format; listed APIs, hosting and discovery services have separate terms.

Interfaces Website

Provider APIs.json project

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Applebot — Publisher Guide

Understand Apple's crawler, search discovery and separate controls for training and AI answer context.

Crawl & indexingFree
Details

Understand Apple's crawler, search discovery and separate controls for training and AI answer context.

Owner next step Review crawler identity and public-page access, then choose search, training and answer-context controls separately.

Access Public documentation or dashboard; no account needed to read

Cost Free. Free public documentation. Crawl access does not guarantee search placement or referrals.

Interfaces Website

Provider Apple

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Applebot
  • Crawler controls

Bing — Verify Bingbot

Check whether an IP address belongs to Bingbot rather than trusting the crawler name supplied in an HTTP user-agent string.

Crawl & indexingFree
Details

Check whether an IP address belongs to Bingbot rather than trusting the crawler name supplied in an HTTP user-agent string.

Owner next step Choose an address from logs you control and follow Microsoft's verification method before interpreting it as genuine Bingbot traffic or changing crawl controls.

Access Public tool or Bing Webmaster Tools account; an IP address from your own logs

Cost Free. Public verification guidance and a free webmaster tool. The public tool may require a human verification challenge; an account provides another route. Verified identity does not establish indexing or referrals.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • Bingbot
  • Bot identity
  • Bot traffic

Bing Webmaster Tools — AI Performance

See where your site is cited across supported Microsoft AI experiences and explore citation trends.

AI visibilityCheck terms
Details

See where your site is cited across supported Microsoft AI experiences and explore citation trends.

Owner next step Verify your site in Bing Webmaster Tools, then review AI Performance and cited pages.

Access Account and verified site

Cost Check current provider terms. Check current access terms. AI Performance features are in preview.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-01

  • AI citations
  • Measurement
  • Site owners

Bing Webmaster Tools — Crawl Control

Configure an hourly Bingbot crawl pattern to balance discovery work with the periods when a site is serving more visitors.

Crawl & indexingFree
Details

Configure an hourly Bingbot crawl pattern to balance discovery work with the periods when a site is serving more visitors.

Owner next step Inspect genuine crawl activity and server load first. If adjustment is justified, review the time zone and hourly pattern for your own site before saving it.

Access Bing Webmaster Tools account with access to the selected site

Cost Free. Part of Bing's webmaster tools. A robots.txt crawl-delay takes precedence over this setting. This controls Bingbot's crawl pattern, not other AI bots, guaranteed crawl volume or search inclusion.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • Bingbot
  • Crawl scheduling
  • Bot traffic

Bing Webmaster Tools — Robots.txt Tester

Test a URL against robots rules, identify the matching allow or disallow statement, and review edits before updating your site.

Crawl & indexingFree
Details

Test a URL against robots rules, identify the matching allow or disallow statement, and review edits before updating your site.

Owner next step Test a representative public URL with the appropriate user agent. Review the proposed rules before separately deploying an approved change.

Access Bing Webmaster Tools account and access to the selected site

Cost Free. Part of the free webmaster toolkit. An editor test does not deploy your changes, establish a security boundary or guarantee search inclusion.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Robots rules
  • Bingbot
  • Crawler access

Bing Webmaster Tools — Site Explorer

Explore a site’s crawled URL folders and recent crawl information to find discovery and indexing issues across groups of pages.

Crawl & indexingFree
Details

Explore a site’s crawled URL folders and recent crawl information to find discovery and indexing issues across groups of pages.

Owner next step Review the folders that matter to your audience, investigate affected URLs and use the linked inspection tools for specific problems.

Access Bing Webmaster Tools account and added site

Cost Free. Part of the free webmaster toolkit; account and site requirements apply. Crawl observations are not proof of AI citations or useful visits.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Crawl diagnostics
  • Site structure
  • Bingbot

Bing Webmaster Tools — Site Scan

Use an on-demand technical site audit to identify common crawl and search-readiness issues in Bing Webmaster Tools.

Crawl & indexingFree
Details

Use an on-demand technical site audit to identify common crawl and search-readiness issues in Bing Webmaster Tools.

Owner next step Add your site to Webmaster Tools, review current scan limits and inspect issues on important public pages before changing the site.

Access Bing Webmaster Tools account and added site

Cost Free. Part of the free webmaster toolkit; account, site and scan-quota requirements apply. A clean scan does not guarantee indexing or AI answer inclusion.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Technical SEO
  • Crawl errors
  • Site audit

Bing Webmaster Tools — URL Inspection

Investigate an added site’s page status in Bing with index, SEO and markup details and a live URL check.

Crawl & indexingFree
Details

Investigate an added site’s page status in Bing with index, SEO and markup details and a live URL check.

Owner next step Select a URL from your added site, inspect its index and crawl details, and compare the live response before changing access rules.

Access Bing Webmaster Tools account and added site

Cost Free. Part of the free webmaster toolkit; account, site and tool limits apply. Crawl eligibility is separate from indexing, AI citations and referral visits.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Bingbot
  • URL inspection
  • Indexing

Brave Search — Crawler Guide

Explore Brave's crawling and indexing guidance for the search index behind its search and AI answer features.

Crawl & indexingFree
Details

Explore Brave's crawling and indexing guidance for the search index behind its search and AI answer features.

Owner next step Review Googlebot crawl eligibility and noindex behavior in Brave's guide before changing public-page access.

Access Public documentation or dashboard; no account needed to read

Cost Free. Free public guidance. Search discovery is separate from paid API access; crawl eligibility does not guarantee AI citations.

Interfaces Website

Provider Brave

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Brave Search
  • Indexing

Browserbase Documentation

Explore public browser platform guides, API references and operational documentation through an agent-readable index.

AI-readable sitesFree
Details

Explore public browser platform guides, API references and operational documentation through an agent-readable index.

Owner next step Study how public capabilities, prerequisites and billing guides are organized before documenting your own service.

Access Public documentation; authenticated operations require separate authorization

Cost Free. Public documentation is free to read. Cloud browser sessions, compute and connected operations have separate plans, permissions and usage charges.

Interfaces Website · Markdown

Provider Browserbase

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Cloudflare — Web Bot Auth Guide

Explore cryptographically signed HTTP requests and Cloudflare's verification workflow for identifying participating bots and agents.

Agent connectionsFree
Details

Explore cryptographically signed HTTP requests and Cloudflare's verification workflow for identifying participating bots and agents.

Owner next step Read the signing, public-key directory and registration requirements before implementing an agent identity. Verified identity still needs an intentional access policy.

Access Public guide; technical implementation and bot registration for verification

Cost Free. Free public documentation. Implementation, hosting and associated Cloudflare features have their own requirements and costs; verification is not permission or a ranking signal.

Interfaces Website · HTTP

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-18

  • Bot operators
  • Site owners
  • Request signatures
  • Agent identity

Cloudflare AI Crawl Control

Observe AI crawler activity and manage access to content on a Cloudflare-hosted zone.

Crawl & indexingFree plan
Details

Observe AI crawler activity and manage access to content on a Cloudflare-hosted zone.

Owner next step Inspect request patterns and existing crawl policies. Decide which crawlers to allow for your goals.

Access Cloudflare account and managed zone

Cost Free plan. Available on all Cloudflare plans; some analytics and related features require a paid plan.

Interfaces Website

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • Bot traffic
  • Crawler controls
  • Analytics

Cloudflare Documentation

Explore public product documentation with an agent navigation index and documented Markdown representations.

AI-readable sitesFree
Details

Explore public product documentation with an agent navigation index and documented Markdown representations.

Owner next step Compare its topic indexes and readable page structure with the documentation entry points on your own site.

Access Public documentation

Cost Free. Public documentation is free; Cloudflare products, plans and operations have separate terms.

Interfaces Website · Markdown

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Cloudflare Markdown for Agents

Serve Markdown representations of supported HTML pages when agents request a documented content type.

Agent connectionsCheck terms
Details

Serve Markdown representations of supported HTML pages when agents request a documented content type.

Owner next step Review plan eligibility and conversion limits, then assess whether a Markdown representation would make your public content clearer.

Access Eligible Cloudflare zone and configuration permissions

Cost Check current provider terms. Included without a separate feature charge on eligible Pro, Business, Enterprise and SSL for SaaS plans; an eligible plan may cost money.

Interfaces HTTP · Markdown

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Cloudflare MCP Servers

Explore official remote MCP services for interacting with Cloudflare infrastructure and information.

Agent connectionsCheck terms
Details

Explore official remote MCP services for interacting with Cloudflare infrastructure and information.

Owner next step Review the relevant server, requested permissions and product costs before authorizing a connection.

Access MCP-compatible client; authorization depends on server

Cost Check current provider terms. Public documentation is free; connected products and operations have separate terms.

Interfaces Website · MCP

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

Cloudflare Radar — AI Insights

Explore aggregate AI crawler activity, crawl purposes and crawl-to-referral trends across Cloudflare's observed traffic.

AI visibilityFree
Details

Explore aggregate AI crawler activity, crawl purposes and crawl-to-referral trends across Cloudflare's observed traffic.

Owner next step Use the public trends as context, then compare them with your own server logs, citations and referral analytics.

Access Public documentation or dashboard; no account needed to read

Cost Free. Public dashboard browsing is free. Aggregate observations do not measure your site's traffic or prove a provider will send visitors.

Interfaces Website

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • AI crawler trends
  • Referral measurement

Cloudflare Radar — Bots Directory

Look up bots and agents listed by Cloudflare to understand their operators, stated purposes and available identity information.

Crawl & indexingFree
Details

Look up bots and agents listed by Cloudflare to understand their operators, stated purposes and available identity information.

Owner next step Look up the crawler you observe, then verify requests with the operator's current guidance before choosing an access policy.

Access Public directory; no account required to browse

Cost Free. Free public directory. A directory entry does not authenticate an individual request, certify a bot or prove indexing, citations or referrals.

Interfaces Website

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Bot identity
  • Crawler reference

Cloudflare Web Analytics

Review page views, referrers and website performance as context for visits from AI experiences.

AI visibilityFree
Details

Review page views, referrers and website performance as context for visits from AI experiences.

Owner next step Inspect available referrers alongside your server logs; keep human visits, bot requests and citations separate.

Access Account and website integration

Cost Free. Free web analytics; browser measurements do not represent all bot requests.

Interfaces Website

Provider Cloudflare

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

Common Crawl — CCBot Guide

Identify Common Crawl requests and review controls for inclusion in its open repository of web crawl data.

Crawl & indexingFree
Details

Identify Common Crawl requests and review controls for inclusion in its open repository of web crawl data.

Owner next step Verify crawler identity with the official IP information and review reuse implications before allowing public content to be collected.

Access Public documentation or dashboard; no account needed to read

Cost Free. Free public documentation. Inclusion in an open crawl dataset is not an AI answer citation or a traffic promise.

Interfaces Website

Provider Common Crawl Foundation

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Open web data
  • CCBot

Context7

Make version-specific library documentation available to AI coding assistants through a documented MCP integration.

Agent connectionsCheck terms
Details

Make version-specific library documentation available to AI coding assistants through a documented MCP integration.

Owner next step Read the library-owner guide to assess whether your public software documentation fits the service.

Access MCP-compatible client; API key and limits depend on use

Cost Check current provider terms. Check current plans and rate limits; public documentation is accessible without executing tools.

Interfaces Website · MCP

Provider Upstash

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

DuckAssistBot — AI Answer Crawler Guide

Read DuckDuckGo's guidance for the crawler that retrieves sources for AI-assisted search answers.

Crawl & indexingFree
Details

Read DuckDuckGo's guidance for the crawler that retrieves sources for AI-assisted search answers.

Owner next step Check the published crawler IP information and decide whether your public pages should be potential AI-answer sources.

Access Public documentation or dashboard; no account needed to read

Cost Free. Free public documentation. DuckDuckGo says this crawler's data is not used to train AI models; opting out does not change organic search rankings.

Interfaces Website

Provider DuckDuckGo

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • AI answers
  • Crawler controls

Exa Documentation

Explore public documentation and integration guides for search and web information APIs through an agent-readable index.

AI-readable sitesFree
Details

Explore public documentation and integration guides for search and web information APIs through an agent-readable index.

Owner next step Use the documentation index as an example of making search capabilities and integration requirements easy to find.

Access Public documentation; account operations are separate

Cost Free. Public documentation is free to read. Search and other API operations have separate pricing, account requirements and usage limits.

Interfaces Website · Markdown

Provider Exa

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Firecrawl Documentation

Explore public guides and API references for a web data service through a Markdown documentation index.

AI-readable sitesFree
Details

Explore public guides and API references for a web data service through a Markdown documentation index.

Owner next step Study how public API capabilities and guides are linked; review usage limits before considering a service integration.

Access Public documentation; account operations are separate

Cost Free. Public documentation is free to read. Crawling, extraction and API execution have separate usage limits and pricing.

Interfaces Website · Markdown

Provider Firecrawl

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Gateway Agent Payment API

Discover Gateway agent services and integration contracts on the dedicated wallet API origin.

Agent connectionsCheck terms
Details

Discover Gateway agent services and integration contracts on the dedicated wallet API origin.

Owner next step Read discovery and contract metadata. Payment needs the wallet owner’s authorization.

Access Public catalog; private purchase capability and approved payment for execution

Cost Check current provider terms. Public discovery is free; service execution is paid. Current quotes and availability are authoritative.

Interfaces REST · OpenAPI · x402

Provider Gateway Information Group

Source evidence Official public source · Source checked 2026-10-01

  • Agent commerce
  • x402
  • OpenAPI

Gateway Information Group

Public information and services with readable product pages, structured discovery and dedicated agent integration guides.

AI-readable sitesCheck terms
Details

Public information and services with readable product pages, structured discovery and dedicated agent integration guides.

Owner next step Explore the readable pages and linked contracts as an example of describing your own site clearly.

Access Public site; service execution has separate requirements

Cost Check current provider terms. Public browsing and discovery are free. Products and services have their own prices and access terms.

Interfaces Website · REST · OpenAPI · Markdown

Provider Gateway Information Group

Source evidence Official public source · Source checked 2026-10-01

  • Business site
  • Structured discovery
  • Public documentation

Gateway Product Discovery API

Read Gateway product contracts, delivery information and current purchase availability as structured JSON.

Agent connectionsFree
Details

Read Gateway product contracts, delivery information and current purchase availability as structured JSON.

Owner next step Use this as an example of clear capability, contract and availability metadata.

Access Anonymous read-only GET

Cost Free. Discovery is free. Listed services and products may be paid; refresh purchase availability before a quote.

Interfaces REST · OpenAPI

Provider Gateway Information Group

Source evidence Official public source · Source checked 2026-10-01

  • JSON
  • OpenAPI
  • Discovery

GitHub MCP Server

Explore GitHub’s official MCP server for connecting assistants to repository and platform capabilities.

Agent connectionsCheck terms
Details

Explore GitHub’s official MCP server for connecting assistants to repository and platform capabilities.

Owner next step Read supported toolsets and authentication requirements; choose the minimum permissions needed.

Access MCP-compatible client; GitHub authentication for supported operations

Cost Check current provider terms. Public source and documentation are available; GitHub plan and hosting terms apply.

Interfaces Website · MCP

Provider GitHub

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

Glama MCP Directory

Find MCP servers, connectors and clients with provider-specific listing and ownership processes.

API & server registriesCheck terms
Details

Find MCP servers, connectors and clients with provider-specific listing and ownership processes.

Owner next step Review your official repository or endpoint documentation and the current claiming requirements before listing.

Access Public browsing; account and ownership checks for managing listings

Cost Check current provider terms. Directory browsing and hosting are different activities; check current service and listing terms.

Interfaces Website

Provider Glama

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

GoAccess — Web Log Analyzer

Analyze server access logs with terminal or HTML reports for requested pages, status codes, referrers and user agents, including a crawler-only view.

Crawl & indexingFree
Details

Analyze server access logs with terminal or HTML reports for requested pages, status codes, referrers and user agents, including a crawler-only view.

Owner next step Confirm your log format, analyze a limited sample on your own system and review crawler requests. Keep reports containing visitor details private.

Access Supported server/terminal environment, log access and technical setup

Cost Free. Free MIT-licensed software. Server access, setup and hosting are separate; a user-agent label or visitor estimate is not verified identity or proof of a human visit.

Interfaces Website

Provider GoAccess project

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Bot traffic
  • Server logs
  • Open source

Google — JavaScript SEO Guide

Understand how Google crawls, renders and indexes JavaScript pages, and diagnose content or links that are absent from the initial response.

Crawl & indexingFree
Details

Understand how Google crawls, renders and indexes JavaScript pages, and diagnose content or links that are absent from the initial response.

Owner next step Compare initial HTML with rendered content, check crawlable links and required resources, and test an important page with the appropriate inspection tool.

Access Public documentation; site-source or developer access for implementation

Cost Free. Free technical documentation. Implementation and hosting are separate. Google's rendering behavior is not a promise that every AI crawler executes JavaScript; no indexing, citation or traffic guarantee applies.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • JavaScript
  • Rendering
  • Crawlable content

Google — Product Structured Data Guide

Choose product-snippet or merchant-listing markup so public product pages expose appropriate price, availability and other supported details in structured form.

AI visibilityFree
Details

Choose product-snippet or merchant-listing markup so public product pages expose appropriate price, availability and other supported details in structured form.

Owner next step Choose the markup type matching your page, keep fields consistent with visible product information, and validate a representative page before deployment.

Access Public guide; accurate product information and the ability to edit your own page markup

Cost Free. Free documentation and markup guidance; development and commerce services are separate. Merchant-listing eligibility has additional requirements. Valid markup does not guarantee rich results, AI citations or sales.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • Product discovery
  • Structured data
  • Merchant metadata

Google — Verify Crawler Requests

Use Google's published IP ranges or matching reverse and forward DNS checks to verify crawler requests rather than trusting a user-agent label alone.

Crawl & indexingFree
Details

Use Google's published IP ranges or matching reverse and forward DNS checks to verify crawler requests rather than trusting a user-agent label alone.

Owner next step Choose a request from your own logs and follow the documented verification method for that crawler or fetcher category before treating it as genuine.

Access Public documentation; server logs and suitable network tools for your own verification

Cost Free. Free public guidance and IP-range references. Log access and technical setup are separate; verified identity does not prove indexing, citations or useful visits.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Googlebot
  • Bot identity
  • Crawler verification

Google AI Search — Owner Guide

Official guidance for making website content eligible for AI features in Google Search, including crawl access and content quality.

AI visibilityFree
Details

Official guidance for making website content eligible for AI features in Google Search, including crawl access and content quality.

Owner next step Check indexing, crawl access, useful text and accurate structured data before looking for special AI optimizations.

Access Public documentation

Cost Free. Free public documentation; inclusion in AI answers is not guaranteed.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-01

  • AI search
  • Discoverability
  • SEO

Google Analytics — Referral Measurement Guide

Use the GA4 Traffic acquisition report and session source / medium to investigate visits from AI assistants and other referring sites.

AI visibilityFree
Details

Use the GA4 Traffic acquisition report and session source / medium to investigate visits from AI assistants and other referring sites.

Owner next step Review session sources and landing pages, then compare engagement and key events. Treat missing referral information as an attribution limit, not proof of no AI visits.

Access Public guide; property access and collected data to use the report

Cost Free. Free public documentation. Using the report requires an Analytics property, measurement setup and appropriate permissions; separate service and privacy requirements apply.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Referral measurement
  • Analytics
  • GA4

Google Crawler Reference

Understand Google crawler and user-triggered fetcher identities, verification and robots.txt behavior.

Crawl & indexingFree
Details

Understand Google crawler and user-triggered fetcher identities, verification and robots.txt behavior.

Owner next step Compare server-log requests with official verification guidance before changing crawler access.

Access Public documentation

Cost Free. Reference documentation is free; hosting and log tooling have separate costs.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Crawler access
  • Documentation

Google Merchant Center — Free Product Listings

Make eligible product pages available for unpaid discovery across supported Google surfaces, which can include Gemini, Search and Shopping.

AI visibilityFree
Details

Make eligible product pages available for unpaid discovery across supported Google surfaces, which can include Gemini, Search and Shopping.

Owner next step Review merchant and product eligibility, verify your store, and keep product data, availability, shipping and return information accurate before enabling free listings.

Access Merchant Center account, eligible store and compliant product data

Cost Free. Eligible free listings have no placement charge. Merchant policies, supported markets and product requirements apply; ads are separate and display is not guaranteed.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Product discovery
  • Gemini
  • Free listings

Google Rich Results Test

Test supported structured data on a public page or pasted code for Google rich-result eligibility.

AI visibilityFree
Details

Test supported structured data on a public page or pasted code for Google rich-result eligibility.

Owner next step Check important pages and correct errors while keeping markup consistent with visible content.

Access Public test tool

Cost Free. Free test; it does not validate every Schema.org type or guarantee search appearance.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

Google robots.txt Guide

Understand how robots.txt controls crawling and why crawl restrictions differ from indexing and private-content protection.

Crawl & indexingFree
Details

Understand how robots.txt controls crawling and why crawl restrictions differ from indexing and private-content protection.

Owner next step Review accidental crawl blocks on useful public pages; use appropriate access controls for private content.

Access Public documentation

Cost Free. Free documentation; robots.txt is not an access-control mechanism.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Google Search Console

Review indexing and search performance, including impressions in Google's generative AI Search report when sufficient data is available.

AI visibilityFree
Details

Review indexing and search performance, including impressions in Google's generative AI Search report when sufficient data is available.

Owner next step Verify your property and inspect important pages, then review generative AI impressions separately from referral visits and conversions.

Access Google account and verified property; enough AI impressions for the report to appear

Cost Free. Free Search Console reports. The generative AI Search report measures impressions, not referral visits, conversions or every AI platform.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Discovery
  • Measurement
  • AI impressions

Google Search Console — Crawl Stats

Review Google's crawl requests, response codes, download sizes and host availability to investigate discovery problems across a site.

Crawl & indexingFree
Details

Review Google's crawl requests, response codes, download sizes and host availability to investigate discovery problems across a site.

Owner next step Use an eligible root-level property to inspect host availability and response trends, then investigate affected URLs before changing crawl rules.

Access Search Console access to an eligible root-level Domain or URL-prefix property

Cost Free. Part of Search Console. This advanced report is intended for larger sites; crawl activity is not proof of indexing, AI citations or referral visits.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Googlebot
  • Bot traffic
  • Host availability

Google Search Console — Sitemaps Report

Submit a public sitemap for a Search Console property and review its processing history, fetch failures and parsing errors.

Crawl & indexingFree
Details

Submit a public sitemap for a Search Console property and review its processing history, fetch failures and parsing errors.

Owner next step Check that your sitemap contains useful canonical pages and is publicly readable, then review its processing status in the correct property.

Access Search Console account with the required property permissions and a public sitemap

Cost Free. Part of Search Console; property permissions apply. Sitemap submission helps discovery but does not guarantee crawling, indexing or AI citations.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Sitemaps
  • Discovery
  • Indexing

Google Search Console — URL Inspection

Inspect a page’s indexed status, crawl access and selected canonical URL, or run a live test to investigate discovery problems.

Crawl & indexingFree
Details

Inspect a page’s indexed status, crawl access and selected canonical URL, or run a live test to investigate discovery problems.

Owner next step Inspect an important URL in your verified property. Compare indexed data with a live test before requesting another crawl.

Access Search Console access to the property containing the URL

Cost Free. Part of Search Console with property and daily request limits. A successful test does not guarantee indexing, search appearance or AI citations.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • URL inspection
  • Indexing
  • Canonical URLs

HubSpot — AI Search Grader

Get a one-time snapshot of how supported AI systems describe a brand, with a written interpretation of its model-generated brand perception scores.

AI visibilityFree
Details

Get a one-time snapshot of how supported AI systems describe a brand, with a written interpretation of its model-generated brand perception scores.

Owner next step Use public brand details, review the report-access form and treat the results as a snapshot to check against actual citations and referral evidence.

Access Public brand details and a report-access form; the provider states no account is required

Cost Free. The one-time grader is free; a form is required for the full report. Ongoing HubSpot AEO monitoring is a separate paid product. Scores are not measured referral traffic.

Interfaces Website

Provider HubSpot

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI search
  • Brand perception
  • One-time report

Hugging Face MCP Server

Discover models, datasets, Spaces and documentation through Hugging Face’s official MCP integration.

Agent connectionsCheck terms
Details

Discover models, datasets, Spaces and documentation through Hugging Face’s official MCP integration.

Owner next step Review account settings and enabled tools; separate discovery from paid compute or write operations.

Access MCP-compatible client and account settings

Cost Check current provider terms. Public documentation is free; hosted tools, jobs and compute may have charges.

Interfaces Website · MCP

Provider Hugging Face

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

IndexNow

Notify participating search engines when public pages are added, updated or removed.

Crawl & indexingCheck terms
Details

Notify participating search engines when public pages are added, updated or removed.

Owner next step Use an existing supported integration or follow the official implementation guide for your site.

Access Website verification key for submissions

Cost Check current provider terms. Read implementation requirements and provider terms. Notification does not guarantee indexing.

Interfaces HTTP

Provider IndexNow participating search engines

Source evidence Official public source · Source checked 2026-10-01

  • Fresh content
  • Indexing
  • Site owners

Known Agents — Agent Analytics

Measure visits from AI agents and crawlers separately from human activity, then investigate which public pages automated visitors request.

AI visibilityFree plan
Details

Measure visits from AI agents and crawlers separately from human activity, then investigate which public pages automated visitors request.

Owner next step Review the visit-event integration and privacy implications for a site you control, then compare crawler activity with actual AI referrals before changing access rules.

Access Account, project setup and a supported visit-event integration on your own site

Cost Free plan. The documented free plan covers one project and 100K monthly events with basic features and a hard limit. API requests and batched items count toward usage; paid plans add capacity and features.

Interfaces Website

Provider Known Agents (formerly Dark Visitors)

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • AI referrals
  • Agent analytics
  • Bot traffic

Known Agents — AI Bot Robots.txt Checker

Review which catalogued AI agents and search crawlers a public site's robots.txt allows or blocks, with explanations of the different bot categories.

Crawl & indexingFree
Details

Review which catalogued AI agents and search crawlers a public site's robots.txt allows or blocks, with explanations of the different bot categories.

Owner next step Check a public site you control and review the relevant search and user-directed agents individually. Decide on rules before separately deploying a change.

Access Public checker accepting a website URL

Cost Free. Listed as a free public tool. Provider analytics and automatic rule management are separate services. Robots rules are not authentication or a guarantee of compliance, citations or traffic.

Interfaces Website

Provider Known Agents

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • Robots rules
  • AI crawlers
  • Bot traffic

LangChain Documentation

Explore public agent development documentation with Markdown topic indexes, package guides and API-description links.

AI-readable sitesFree
Details

Explore public agent development documentation with Markdown topic indexes, package guides and API-description links.

Owner next step Use its topic-based indexes as an example for organizing complex public documentation into useful entry points.

Access Public documentation; authenticated operations require separate authorization

Cost Free. Public documentation is free to read. Hosted products, deployments, compute and connected model providers have separate terms and costs.

Interfaces Website · Markdown

Provider LangChain

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

llms.txt — Agent Navigation Proposal

A proposed Markdown convention for giving agents concise background and links to useful site documentation.

Agent connectionsFree
Details

A proposed Markdown convention for giving agents concise background and links to useful site documentation.

Owner next step Describe your public site accurately and link to useful documentation; keep robots.txt and access policies separate.

Access Public documentation

Cost Free. Free public specification. It is a proposal, not a crawler permission rule or a traffic guarantee.

Interfaces Markdown

Provider llms.txt project

Source evidence Official public source · Source checked 2026-10-01

  • Agent navigation
  • Markdown
  • Site owners

Matomo On-Premise Analytics

Use self-hosted website analytics to examine referral visits and useful visitor actions.

AI visibilityFree
Details

Use self-hosted website analytics to examine referral visits and useful visitor actions.

Owner next step Evaluate deployment and privacy requirements, then compare referral traffic with your own crawler evidence.

Access Self-hosted installation and website integration

Cost Free. Self-hosted software is free; infrastructure, optional plugins and hosted services may cost extra.

Interfaces Website

Provider Matomo

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

MCP Apps — Interactive Client Interfaces

Use the MCP Apps extension to associate server tools with interactive HTML interfaces displayed by supporting MCP hosts.

Agent connectionsFree
Details

Use the MCP Apps extension to associate server tools with interactive HTML interfaces displayed by supporting MCP hosts.

Owner next step Check client support, prototype a narrow tool and UI resource, and review host permissions and content security policy before distributing your service.

Access MCP server development and a host supporting the Apps extension

Cost Free. Free public protocol documentation. Server hosting, client access and connected services have their own costs and permissions. An MCP App is not automatic publication in a host's directory or authorization to execute tools.

Interfaces Website · MCP

Provider Model Context Protocol project

Source evidence Official public source · Source checked 2026-10-20

  • Agent developers
  • MCP Apps
  • Interactive tools
  • Client support

MCP Inspector

Test and debug an MCP server's connection and exposed capabilities using the reference browser, command-line or terminal developer clients.

Agent connectionsFree
Details

Test and debug an MCP server's connection and exposed capabilities using the reference browser, command-line or terminal developer clients.

Owner next step Review the current runtime and release requirements, then test your own server's advertised tools and authentication before sharing its discovery page.

Access Local Node.js setup and authorization for the server and operations being tested

Cost Free. Free open-source developer tooling; local setup and runtime requirements apply. Connected services retain their own permissions and charges; testing can execute server tools.

Interfaces Website · MCP

Provider Model Context Protocol project

Source evidence Official public source · Source checked 2026-10-18

  • Agent developers
  • MCP
  • Server testing
  • Authentication

Microsoft Clarity — AI Visibility

Explore citations in AI answers and compare them with AI referral sessions. Optional bot-activity reporting uses a supported edge-provider connection.

AI visibilityFree
Details

Explore citations in AI answers and compare them with AI referral sessions. Optional bot-activity reporting uses a supported edge-provider connection.

Owner next step Verify your domain, establish a citation baseline, then compare cited pages with referral sessions and useful visitor actions.

Access Clarity account, project and verified domain; extra setup for bot activity

Cost Free. Clarity is free. Citation reporting requires domain verification; bot activity requires a supported CDN or edge integration, whose own costs and terms still apply.

Interfaces Website

Provider Microsoft

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Referral measurement

Mintlify Documentation

Explore public documentation for building and maintaining documentation websites, with a Markdown index for agents.

AI-readable sitesFree
Details

Explore public documentation for building and maintaining documentation websites, with a Markdown index for agents.

Owner next step Compare its documentation navigation and public guides with the structure of your own site.

Access Public documentation; account operations are separate

Cost Free. Public documentation is free to read. Documentation hosting, AI features and API operations have separate plans and terms.

Interfaces Website · Markdown

Provider Mintlify

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Mistral — Crawler and Search Access Guide

Review Mistral's separate user-request, search-indexing and model-training agents, with published user-agent and IP information.

Crawl & indexingFree
Details

Review Mistral's separate user-request, search-indexing and model-training agents, with published user-agent and IP information.

Owner next step Compare MistralAI-User, MistralAI-Index and MistralAI-Training in the current guide before choosing separate access rules for public pages.

Access Public documentation

Cost Free. Free public documentation. Allowing a search or user-request agent does not guarantee citations or visitor traffic.

Interfaces Website

Provider Mistral AI

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Mistral
  • Crawler controls
  • Search indexing

Neon Documentation

Explore public database documentation and guides through a Markdown navigation index.

AI-readable sitesFree
Details

Explore public database documentation and guides through a Markdown navigation index.

Owner next step Study the organization of public database guides and integration information before designing your own documentation entry point.

Access Public documentation; account operations are separate

Cost Free. Public documentation is free to read. Database hosting, storage, compute and administrative operations have separate plans and requirements.

Interfaces Website · Markdown

Provider Neon

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

NLWeb — Conversational Site Interfaces

Explore open protocols and reference code for conversational website interfaces that use structured content and can also serve MCP clients.

Agent connectionsFree
Details

Explore open protocols and reference code for conversational website interfaces that use structured content and can also serve MCP clients.

Owner next step Evaluate the reference implementation with a small public content set, then review retrieval quality, hosting, model dependencies and permissions before publishing an endpoint.

Access Developer setup, suitable content and separately configured hosting/model services

Cost Free. Public MIT-licensed reference code and documentation. Hosting, model inference and retrieval services may incur charges; this is not a free managed deployment or a guarantee of agent traffic.

Interfaces Website · MCP

Provider NLWeb project

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • Agent developers
  • NLWeb
  • Structured content

Official MCP Registry

A metadata repository for public MCP servers with namespace ownership checks and a REST discovery interface.

API & server registriesCheck terms
Details

A metadata repository for public MCP servers with namespace ownership checks and a REST discovery interface.

Owner next step Prepare server metadata and verify your namespace using the official publishing guide.

Access Public metadata; namespace verification for publishing

Cost Check current provider terms. Review current publishing requirements and terms; the registry is in preview.

Interfaces REST · OpenAPI

Provider Model Context Protocol project

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Server discovery
  • Site owners

OpenAI Search and Crawler Guide

Distinguish search crawling, model-training crawling and user-triggered page access using official bot documentation.

Crawl & indexingFree
Details

Distinguish search crawling, model-training crawling and user-triggered page access using official bot documentation.

Owner next step Review OAI-SearchBot and GPTBot separately against your discovery and content-use preferences.

Access Public documentation

Cost Free. Free guidance; crawler access does not guarantee inclusion or citations.

Interfaces Website

Provider OpenAI

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Crawler access
  • Documentation

OpenAPI Specification

Describe an HTTP API with a machine-readable contract covering operations, inputs, responses and security requirements.

Agent connectionsFree
Details

Describe an HTTP API with a machine-readable contract covering operations, inputs, responses and security requirements.

Owner next step Document your actual public API and validate a supported specification version before sharing it with directories or agents.

Access Public specification

Cost Free. Free public specification; API implementation, hosting and service execution have separate costs.

Interfaces Website · OpenAPI

Provider OpenAPI Initiative

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

OtterlyAI

Track selected AI-search prompts, brand visibility and citation sources, with reports for comparing changes over time.

AI visibilityCheck terms
Details

Track selected AI-search prompts, brand visibility and citation sources, with reports for comparing changes over time.

Owner next step Define a small set of relevant prompts and compare citation patterns with real referral outcomes before choosing a plan or expanding coverage.

Access Provider account and plan; current trial terms may apply

Cost Check current provider terms. Paid subscription with prompt and platform limits. A trial may be available; it is not an ongoing free plan. Some engines and integrations require add-ons or higher plans.

Interfaces Website

Provider OtterlyAI

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Brand monitoring
  • Paid service

PageSpeed Insights

Review laboratory performance diagnostics and available real-user performance data for a public page.

AI visibilityCheck terms
Details

Review laboratory performance diagnostics and available real-user performance data for a public page.

Owner next step Test key public pages and prioritize readable, responsive content; treat performance findings separately from AI citations.

Access Public browser tool

Cost Check current provider terms. Public browser diagnostics; API quotas and current terms apply to automated use.

Interfaces Website

Provider Google

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Peec AI

Explore AI-search visibility, competitive presence and cited sources across selected prompts and supported models.

AI visibilityCheck terms
Details

Explore AI-search visibility, competitive presence and cited sources across selected prompts and supported models.

Owner next step Choose prompts that match your audience and inspect source patterns. Compare sampled visibility with your own referral and conversion evidence.

Access Provider account, project and supported plan

Cost Check current provider terms. Paid service priced by tracked prompts and models. A limited trial is advertised; ongoing tracking is not listed here as free. Review current model and project limits.

Interfaces Website

Provider Peec AI

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Competitive research
  • Paid service

Perplexity Search and Crawler Guide

Understand Perplexity search crawling and user-requested fetches, including published bot identities.

Crawl & indexingFree
Details

Understand Perplexity search crawling and user-requested fetches, including published bot identities.

Owner next step Compare official bot identities and IP information with your logs; choose deliberate access policies.

Access Public documentation

Cost Free. Free documentation; allowing access does not guarantee citations or visits.

Interfaces Website

Provider Perplexity

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Crawler access
  • Documentation

Profound — AI Visibility

Explore answer-engine visibility, citation and competitive insights, with separate agent-traffic analytics and marketing tools.

AI visibilityCheck terms
Details

Explore answer-engine visibility, citation and competitive insights, with separate agent-traffic analytics and marketing tools.

Owner next step Review which engines, prompts and analytics are included, then separate sampled answer visibility from observed site requests and referral outcomes.

Access Provider account and suitable plan; analytics integrations may require setup

Cost Check current provider terms. Commercial platform with trial and enterprise terms. Ongoing tracking, engine coverage and agent usage have plan or credit limits; no ongoing free tier is confirmed here.

Interfaces Website

Provider Profound

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Agent analytics
  • Paid service

PulseMCP

Discover MCP servers through a curated directory with public submission and registry-enrichment processes.

API & server registriesCheck terms
Details

Discover MCP servers through a curated directory with public submission and registry-enrichment processes.

Owner next step Review the directory and current submission requirements for a public MCP service.

Access Public browsing; submission and partner API requirements vary

Cost Check current provider terms. Check current submission and API terms; listings may link to paid services.

Interfaces Website

Provider PulseMCP

Source evidence Official public source · Source checked 2026-10-01

  • MCP
  • Documentation
  • Agent discovery

Resend Documentation

Explore public email API documentation, SDK guides and deliverability resources linked in a Markdown index.

AI-readable sitesFree
Details

Explore public email API documentation, SDK guides and deliverability resources linked in a Markdown index.

Owner next step Compare its API references and prerequisites with your own service documentation; reading guides does not send email.

Access Public documentation; account operations are separate

Cost Free. Public documentation is free to read. Email sending, receiving and connected operations have separate account, domain and plan requirements.

Interfaces Website · Markdown

Provider Resend

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Robots Exclusion Protocol — RFC 9309

Read the formal robots exclusion rules for describing which URL paths cooperating crawlers may request.

Crawl & indexingFree
Details

Read the formal robots exclusion rules for describing which URL paths cooperating crawlers may request.

Owner next step Use the standard alongside each crawler’s guidance when reviewing public crawl rules. Protect confidential pages with authentication.

Access Public specification; technical interpretation and site access for implementation

Cost Free. Free public standard. Robots rules are not access authorization or a security boundary; compliance does not guarantee indexing, citations or referrals.

Interfaces Website

Provider IETF

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Robots rules
  • Crawler controls
  • Web standard

Schema Markup Validator

Extract and check Schema.org structured data from a public page or pasted markup, including syntax mistakes.

AI visibilityCheck terms
Details

Extract and check Schema.org structured data from a public page or pasted markup, including syntax mistakes.

Owner next step Validate your visible organization, product or page markup and investigate errors before publishing changes.

Access Public browser tool

Cost Check current provider terms. Public validator; check current service terms. Validation does not guarantee search features or AI citations.

Interfaces Website

Provider Schema.org community; service provided by Google

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Schema.org

Explore shared structured-data vocabulary for describing organizations, products and public web content.

AI visibilityFree
Details

Explore shared structured-data vocabulary for describing organizations, products and public web content.

Owner next step Choose types that accurately describe visible content; avoid invented reviews or unsupported claims.

Access Public vocabulary

Cost Free. Free public vocabulary; markup does not guarantee AI citations or rich results.

Interfaces Website

Provider Schema.org community

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

Screaming Frog — Log File Analyser

Inspect your own server logs to find crawled URLs, response errors and search or AI bot activity. Complements a site crawl with actual request evidence.

Crawl & indexingFree plan
Details

Inspect your own server logs to find crawled URLs, response errors and search or AI bot activity. Complements a site crawl with actual request evidence.

Owner next step Start with a small log sample, review important URLs and error responses, and compare bot requests with separate citation and referral measurements.

Access Desktop application and access to your server logs

Cost Free plan. Free version analyzes up to 1,000 log events in one project. Larger datasets and multiple projects require a paid licence; this is separate from SEO Spider.

Interfaces Website

Provider Screaming Frog

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • Bot traffic
  • Server logs
  • Crawl errors

Screaming Frog SEO Spider

Crawl website links and inspect titles, metadata, errors and redirects to identify technical discovery problems.

Crawl & indexingFree plan
Details

Crawl website links and inspect titles, metadata, errors and redirects to identify technical discovery problems.

Owner next step Run a bounded crawl on a site you control and review broken links, duplicate metadata and crawl directives.

Access Desktop installation; crawl only sites you are authorized to assess

Cost Free plan. Free version supports up to 500 URLs per crawl. Advanced features and larger crawls require a paid licence.

Interfaces Website

Provider Screaming Frog

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Documentation

Scrunch — AI Visibility

Review AI prompt and citation monitoring, page audits and agent or referral metrics to understand how a brand appears across supported AI platforms.

AI visibilityCheck terms
Details

Review AI prompt and citation monitoring, page audits and agent or referral metrics to understand how a brand appears across supported AI platforms.

Owner next step Set a baseline for relevant prompts and pages, confirm the required integrations, and compare visibility measures with actual useful visits.

Access Provider account and plan; integrations or site access for some features

Cost Check current provider terms. Paid plans with prompt, user and page-audit limits. A limited trial is advertised; it is not an ongoing free plan. Additional capabilities have their own access requirements.

Interfaces Website

Provider Scrunch

Source evidence Official public source · Source checked 2026-10-18

  • Site owners
  • AI citations
  • Page audits
  • Paid service

SE Visible — AI Visibility Monitoring

Monitor a configured brand's mentions, cited sources and competitors across supported AI answer engines and reporting markets.

AI visibilityCheck terms
Details

Monitor a configured brand's mentions, cited sources and competitors across supported AI answer engines and reporting markets.

Owner next step Confirm current engine, market and prompt coverage, review the paid plan limits, and compare a small set of relevant topics with your own business outcomes.

Access Account and a suitable paid subscription after any trial

Cost Check current provider terms. Paid subscription with a time-limited trial; the trial is not an ongoing free plan. Current prompt, project, engine, country and language coverage varies. Sampled AI answers do not prove customer referrals.

Interfaces Website

Provider SE Ranking

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • AI visibility
  • Prompt monitoring
  • Competitor research

Semrush — AI Search Visibility Checker

Get a limited public snapshot of brand mentions and citations across supported AI search platforms to identify questions and pages worth investigating.

AI visibilityFree
Details

Get a limited public snapshot of brand mentions and citations across supported AI search platforms to identify questions and pages worth investigating.

Owner next step Check your domain, inspect the underlying platforms and cited pages, then validate apparent opportunities against your own referral and conversion measurements.

Access Public domain checker; published daily limits apply

Cost Free. The provider documents three free uses per day without an account. Ongoing monitoring and deeper analysis belong to the separate paid toolkit; a visibility score is not measured referral traffic.

Interfaces Website

Provider Semrush

Source evidence Official public source · Source checked 2026-10-20

  • Site owners
  • AI visibility
  • Brand mentions
  • Citations

Sitemaps Protocol

Describe public page locations and supported sitemap metadata using the shared XML sitemap protocol.

Crawl & indexingFree
Details

Describe public page locations and supported sitemap metadata using the shared XML sitemap protocol.

Owner next step List canonical public URLs in a valid sitemap and keep it current as pages change.

Access Public specification

Cost Free. Free specification; sitemap inclusion does not guarantee indexing.

Interfaces Website

Provider Sitemaps.org

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Crawler access
  • Documentation

Smithery — MCP Directory

Browse a public MCP registry and find direct provider endpoints. Smithery is retaining its registry while ending its proxy and connection-management services on October 31, 2026.

API & server registriesCheck terms
Details

Browse a public MCP registry and find direct provider endpoints. Smithery is retaining its registry while ending its proxy and connection-management services on October 31, 2026.

Owner next step Use the official provider MCP URL shown on a registry page, verify its capabilities and permissions, and read the transition notice before relying on a Smithery connection or publication workflow.

Access Public registry browsing; direct provider connections retain their own authentication and permissions

Cost Check current provider terms. Public registry browsing; connected providers set their own access and charges. Smithery is ending connection management on October 31, 2026. Inclusion is not a security certification.

Interfaces Website · MCP

Provider Smithery / Arcade.dev

Source evidence Official public source · Source checked 2026-10-20

  • Agent developers
  • MCP servers
  • Discovery
  • Direct endpoints
  • Service transition

Stripe Documentation

Explore a public documentation site with a Markdown index linking API references, product guides and agent resources.

AI-readable sitesFree
Details

Explore a public documentation site with a Markdown index linking API references, product guides and agent resources.

Owner next step Study the documentation index as an example for organizing your own public API guides. Reading it does not authorize payments or provisioning.

Access Public documentation; authenticated operations are separate

Cost Free. Public documentation is free; Stripe products and transactions have separate costs and requirements.

Interfaces Website · Markdown

Provider Stripe

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Tavily Documentation

Explore public search, extraction and crawling documentation through a Markdown index with links to guides and API references.

AI-readable sitesFree
Details

Explore public search, extraction and crawling documentation through a Markdown index with links to guides and API references.

Owner next step Compare its task-based navigation with your own site's public guides; read usage and pricing requirements before considering API operations.

Access Public documentation; authenticated operations require separate authorization

Cost Free. Public documentation is free to read. Search, extraction, crawling and research operations have separate limits, account requirements and pricing.

Interfaces Website · Markdown

Provider Tavily

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Umami Analytics

Explore an open source, self-hostable web analytics option for measuring visits and referral sources.

AI visibilityCheck terms
Details

Explore an open source, self-hostable web analytics option for measuring visits and referral sources.

Owner next step Review current hosting options and privacy requirements before using referral reports to assess AI-driven visits.

Access Self-hosted or hosted deployment; website integration

Cost Check current provider terms. Open source self-hosting is documented. Check licensing, infrastructure costs and hosted-plan terms.

Interfaces Website

Provider Umami

Source evidence Official public source · Source checked 2026-10-01

  • Site owners
  • Discovery
  • Measurement

Vercel Documentation

Explore public platform documentation with a Markdown agent entry point linking guides, API information and integration resources.

AI-readable sitesFree
Details

Explore public platform documentation with a Markdown agent entry point linking guides, API information and integration resources.

Owner next step Use the documentation index as an example of making public capabilities, prerequisites and related guides easy to find.

Access Public documentation; account operations require separate authorization

Cost Free. Public documentation is free; hosting, compute and connected services may incur charges.

Interfaces Website · Markdown

Provider Vercel

Source evidence Official public source · Source checked 2026-10-01

  • Documentation site
  • Agent navigation
  • Site owners

Website Opportunity Scanner

Explore opportunities to improve a public website, with a report that puts the findings in context.

AI visibilityFree plan
Details

Explore opportunities to improve a public website, with a report that puts the findings in context.

Access Web browser

Cost Free plan. A free three-finding preview. Complete reports and implementation are paid options.

Interfaces Website

Provider Gateway Information Group

Source evidence Official public source · Source check date not recorded

  • Websites
  • Reports

x402 List

A directory of x402 services with a public JSON discovery API, endpoint metadata and integration documentation.

API & server registriesFree plan
Details

A directory of x402 services with a public JSON discovery API, endpoint metadata and integration documentation.

Owner next step Prepare accurate endpoints, payment terms and documentation before suggesting your service.

Access Anonymous read API; paid services have separate terms

Cost Free plan. Directory reads are free. Submission fees can apply under current rules; some assessment tools and listed services are paid.

Interfaces REST · OpenAPI · MCP

Provider x402 List

Source evidence Official public source · Source checked 2026-10-01

  • Agent discovery
  • x402
  • Site submission

A useful place to begin

Turn discovery into a next step

Three practical paths for site owners.
Free guidance. Provider terms still apply.

Understand your AI and bot traffic

Observe requests before deciding which crawlers to allow.

  1. Review crawler activity

    If your site uses Cloudflare, inspect AI crawler requests and current plan limits before changing access.

    Cloudflare AI Crawl Control
  2. Keep discovery intentional

    Check what should be public and how changes are announced. A crawler request is not proof of an AI citation or a human visit.

    IndexNow
  3. Connect signals carefully

    Compare crawler requests with session sources, engagement and key events in your analytics. Referral attribution has limits; bot requests alone are not visits.

    Google Analytics — Referral Measurement Guide
Explore this category →

Give agents a useful introduction

Describe your public site or service clearly before choosing a registry.

  1. Provide readable navigation

    Use concise public documentation and links to useful pages. llms.txt is a navigation proposal, not a crawler permission or ranking guarantee.

    llms.txt — Agent Navigation Proposal
  2. Describe your interface

    Explore a public product-discovery contract as an example. State inputs, access limits and current costs accurately.

    Gateway Product Discovery API
  3. Choose the right registry

    For an MCP server, review namespace and publication requirements. Other API types need a suitable directory and stable documentation.

    Official MCP Registry
Explore this category →

Start with the steps that fit your site. These are editorial guides, not a scan or a guarantee of traffic. Use the owner checklist to review your next steps.

More public resource profiles

Additional references, including Gateway tools and general-purpose software. Read each profile for its scope, source and requirements.

A practical starting point

Prepare your site for discovery

Six things to review before sharing your site.
No account or paid audit required.

0 of 6 items reviewed

This is a personal checklist, not a scan, score or certification. Selections stay in this page session. Search eligibility and good metadata do not guarantee visits or citations. Read Google’s owner guidance.

A little direction

Turn your review into a plan

Prioritize your next steps using the items you have reviewed.

Optional · Free to use
Help me plan what to do next

Selections describe your own review, not test results. This does not scan your site or guarantee visits, citations or rankings. Privacy details.

Prepare your free listing suggestion →

Ready for agents and crawlers

One repository. Open discovery.

Read structured site records without JavaScript or an account. Filter by purpose, interface or free option; use stable listing pages and source evidence.

Make an introduction

Give your site
a clear introduction.

Suggest an AI/bot resource, public site, API or agent service. Explain who it helps, how agents can discover it, and any access or pricing limits. Suggestions are reviewed before they appear.

A little direction

Give your listing a clearer introduction

Get feedback on relevance, clarity and missing access or cost details. Edit the fields yourself before submitting.

Optional · Free to use
Review my listing draft

Fill in the listing name, description and cost notes above to enable draft feedback.

Feedback covers the public text above. Website and integration URLs are excluded. Nothing is saved to the review queue until you submit the form. This does not scan your site or guarantee visits, citations or rankings. Privacy details.

Submit for review saves these details in a private editorial queue. Suggestions do not send email or create a listing immediately. Review timing is not guaranteed. Do not include private data or access tokens. You can also prepare a manual email draft; nothing is emailed until you send it.

Prefer a simple email? Suggest a listing ↗

Opens your email app with a draft. Sending the email requests review; it does not create a listing. You can also email info@gatewayinformationgroup.com.

How review works

  1. A name and public websiteWe check the public destination and who provides it.
  2. A clear, useful descriptionWe look for a practical AI/bot purpose, public documentation, accurate claims and clear access or cost limits.
  3. Your relationship to the listingOwnership is disclosed. Listings are alphabetical by default, with category filters and an optional category sort, with no paid placement or submission fee.

Report an outdated listing or broken link.

A listing gives your site a public description and machine-readable record. It does not guarantee bot visits, AI citations or search rankings. Suggestions are reviewed before they appear. Review does not guarantee inclusion or a response time. Inclusion is an introduction, not a certification or endorsement. Gateway’s own listings are labeled. Do not send passwords, private documents or payment details.