Resources AI Reach Check Methodology
We read each outlet's public robots.txt file and a sample article's page tags, then apply each AI company's own documented rules. No third-party citation data. The bot names come from the open-source ai.robots.txt list, and rules are matched the way the Robots Exclusion Protocol (RFC 9309) specifies.
OAI-SearchBotDisallowed for the article URL. ChatGPT-User (live fetch) is not scored.OpenAI crawler documentationGoogle-ExtendedDisallowed. Google says this token covers Gemini training and grounding in Gemini Apps, not Search.Google crawler documentationGooglebotDisallowed. Rare, since it also removes the outlet from Google Search.Google crawler documentationGooglebotSame rule as AI Overviews.Google crawler documentationClaude-SearchBotDisallowed. Blocking only ClaudeBot opts out of training, not search.Anthropic crawler documentationUser-agent: *The catch-all rule disallows the article URL.xAI publishes no crawler namePerplexityBotDisallowed. Perplexity-User may still fetch pages a user asks for.Perplexity bot documentationBingbot + noarchive tagBingbot is disallowed, or article pages carry a noarchive tag without nocache. Bing keeps noarchive pages out of Copilot answers.Bing Webmaster GuidelinesSample shown: 8 business outlets, robots.txt and article tags read Nov 16, 2026 in a standard browser. Bot list: ai.robots.txt (MIT license). Rules and docs change; every report shows its check date.
Run a check →One issue a week. One AI development that matters for comms, broken down into what it means for your budget, your team, and your next board conversation.