Resources AI Reach Check Methodology

How we check each engine.

We read each outlet's public robots.txt file and a sample article's page tags, then apply each AI company's own documented rules. No third-party citation data. The bot names come from the open-source ai.robots.txt list, and rules are matched the way the Robots Exclusion Protocol (RFC 9309) specifies.

EngineWhat we readCounts as blocked whenBased on
ChatGPTOAI-SearchBotDisallowed for the article URL. ChatGPT-User (live fetch) is not scored.OpenAI crawler documentation
Gemini appGoogle-ExtendedDisallowed. Google says this token covers Gemini training and grounding in Gemini Apps, not Search.Google crawler documentation
AI OverviewsGooglebotDisallowed. Rare, since it also removes the outlet from Google Search.Google crawler documentation
AI ModeGooglebotSame rule as AI Overviews.Google crawler documentation
ClaudeClaude-SearchBotDisallowed. Blocking only ClaudeBot opts out of training, not search.Anthropic crawler documentation
GrokUser-agent: *The catch-all rule disallows the article URL.xAI publishes no crawler name
PerplexityPerplexityBotDisallowed. Perplexity-User may still fetch pages a user asks for.Perplexity bot documentation
CopilotBingbot + noarchive tagBingbot is disallowed, or article pages carry a noarchive tag without nocache. Bing keeps noarchive pages out of Copilot answers.Bing Webmaster Guidelines

How a cell gets its label

  • Open: the engine's bot may fetch a sample article URL.
  • Blocked: a rule naming the bot, or the catch-all rule, disallows that URL. For Copilot, a noarchive tag also counts, unless the page adds nocache.
  • Deal: a licensing or content partnership between the outlet and that AI company, from a public announcement we link to. A deal cell is also open.
  • No opt-out: Grok names no crawler, so only a catch-all block can apply.
  • Reach score: engines not marked Blocked, out of 8.

What this can't tell you

  • Whether an engine actually cites the outlet. Access is the floor, not the result.
  • Blocks made at the network level (firewalls, CDNs) that robots.txt doesn't show.
  • Rules for one section that differ from our sample article URL.
  • Licensing deals that were never announced.
  • How Copilot will treat these outlets after February 1, 2027. Microsoft is replacing noarchive and nocache with new AI Grounding preferences, and the Copilot rule here will change when they take effect.

Sample shown: 8 business outlets, robots.txt and article tags read Nov 16, 2026 in a standard browser. Bot list: ai.robots.txt (MIT license). Rules and docs change; every report shows its check date.

Run a check →

Be the one in the room who saw it coming.

One issue a week. One AI development that matters for comms, broken down into what it means for your budget, your team, and your next board conversation.

A free, weekly, unsubscribe anytime.