OpenAI and Google Back Shared AI Training Opt-Out? How the Robots.txt Signal Works

OpenAI and Google Back Shared AI Training Opt-Out? OpenAI, Google, and Microsoft announced plans on October 30, 2026, to honor a single, shared policy token in robots.txt that allows website operators to express generative AI model training preferences across multiple providers simultaneously. The proposed mechanism originates from an individual Internet-Draft submitted to the IETF AI Preferences working group by Anthropic engineer Kevin Kelley. While the initiative offers publishers a streamlined method to declare training opt-outs using standard Allow and Disallow syntax, implementation timelines remain staggered into 2027, and each major platform plans to interpret and apply the signal differently across model training and live conversational grounding.

Core Industry Realignment: Major AI Labs Coordinate on Robots.txt Standards

At a Glance

• OpenAI, Google, and Microsoft announced coordinated plans on October 30, 2026, to recognize a shared robots.txt policy token, __usage_ai-training, for managing generative model training preferences.
• The signal is defined in an individual IETF Internet-Draft authored by Anthropic engineer Kevin Kelley, establishing a standardized syntax without representing an official or finalized IETF RFC standard.
• Platform adoption schedules diverge significantly: Anthropic has initiated internal engineering work, OpenAI targets rollout in upcoming months, Microsoft begins on February 1, 2027, and Google projects support in 2027.

For nearly thirty years, the Robots Exclusion Protocol governed how automated web crawlers interacted with public online content. Website administrators placed a simple robots.txt plain-text file in their root directory, defining which search engine user-agents had permission to fetch specific URL paths. This protocol operated on a foundational commercial bargain: website publishers permitted crawlers to index their content in exchange for discoverability, outbound referral traffic, and brand exposure on search engine results pages.

The rapid rise of generative artificial intelligence disrupted that economic model. Automated scrapers deployed by frontier AI labs began ingesting web domains to train foundation models, often summarizing original content directly within answer interfaces without delivering referral clicks to publishers. Content owners seeking to prevent their intellectual property from being ingested were forced to maintain fragmented, complex lists of proprietary user-agents, including GPTBot, ClaudeBot, and Google-Extended. This administrative overhead increased the operational risk of configuration errors or accidental search de-indexing when managing multiple crawler directives simultaneously.

The emergence of a shared AI training opt-out marks a significant shift from fragmented vendor blocking to cooperative protocol governance. By coordinating support around a shared policy token, major AI providers aim to lower the technical barrier for content creators who wish to establish clear boundaries regarding generative model training. According to the official Google Search Central announcement, this single rule allows website administrators to declare their generative AI training stance in one step, rather than tracking and configuring distinct directives for every individual commercial AI crawler entering the market.

Under-the-Hood Architectural Mechanics: How the __usage_ai-training Token Works

The technical architecture behind the shared preference signal is specified in the IETF Internet-Draft on Machine-Readable Training Preference Signals. Crucially, the draft does not introduce novel directives or alter the underlying grammar of the Robots Exclusion Protocol defined in RFC 9309. Instead, it reuses the standard User-Agent record by assigning a reserved policy token: __usage_ai-training.

To configure an opt-out, a webmaster creates an ordinary rule block designating this policy token as the user-agent, followed by standard Disallow and Allow rules. The example below illustrates how a website owner expresses an opt-out across all site directories while keeping an exception for public promotional materials:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The semantic definition of what constitutes “AI training” is anchored in the IETF working group’s foundational vocabulary specifications. Under this framework, training is defined strictly as utilizing scraped web content to modify or fine-tune the learned weights and parameters of a generative artificial intelligence model. Non-generative computational operations, such as standard document classification, search index ranking, spelling correction, and spam filtering, are explicitly excluded from this definition.

Protocol Boundaries: Crawling vs. Training vs. Grounding

Understanding the operational limits of this proposed token requires separating the distinct stages of automated data ingestion:

[Public Web Content]
       │
       ▼
[1. HTTP Crawl / Fetch] ──> Governed by specific crawler User-Agents (e.g., GPTBot, Bingbot)
       │                    *__usage_ai-training DOES NOT govern HTTP fetch behavior*
       ├────────────────────────────────────────┐
       ▼                                        ▼
[2. Model Training Use]                  [3. Live Grounding / RAG]
   *__usage_ai-training: Disallow*          *Governed by page tags & vendor settings*
   Signals preference against training      Supplies real-time web context to AI models
       │                                        │
       ▼                                        ▼
   Enforcement depends on                   Eligibility criteria set by
   participating AI providers               individual platform policies

Technical teams must evaluate four structural boundaries inherent in this draft specification:

1. The Signal Governs Training, Not Crawling: The policy token is not a directive for network crawling behavior. An AI crawler encountering Disallow: / under __usage_ai-training is still permitted to fetch the HTML document over HTTP. It is merely instructed not to feed that retrieved text into model training pipelines. To prevent a bot from requesting the URL altogether, administrators must continue to block the crawler’s specific identity name.
2. Wildcard Rules Do Not Apply: General wildcard blocks, such as User-Agent: * followed by Disallow: /, do not match the policy token. The draft mandates that participating AI providers must not interpret a universal wildcard group as an expression of an AI training preference. If a file omits the explicit __usage_ai-training token, the standard interprets the site as expressing no training preference.
3. No Retrospective Model Unlearning: Updating a website’s robots.txt file operates forward in time. It provides a signal for future crawl and training cycles but does not compel an AI lab to purge data already integrated into historical training datasets or delete weights from pre-existing foundation models.
4. Voluntary Coordination Over Technical Enforcement: The signal expresses a publisher preference; it does not provide technical enforcement against non-participating or non-compliant data consumers. Under RFC 9309 principles, robots.txt remains a cooperative advisory mechanism rather than an access-control firewall.

Provider Implementation Differences and Migration Schedules

While Google, Microsoft, and OpenAI have announced plans to support the shared signal, their technical implementations and rollout schedules diverge significantly. Each provider approaches the balance between search engine indexing, generative model training, and conversational retrieval-augmented generation (RAG) through its own product infrastructure.

The matrix below contrasts confirmed platform commitments, interpretation rules, and migration timelines:

Platform	Confirmed Status & Timeline	Policy Token Interpretation	Grounding & Retrieval Handling	Legacy Tag Migration & Exceptions
Google	Announced support; planned for 2027 (exact date TBD)	Interpreted as Google-Extended: Disallow when no explicit Google-Extended rule exists	Default mapping affects Gemini Apps grounding; explicit rules override	Existing Google-Extended directives remain fully valid; search indexing unaffected
Microsoft	Confirmed commitment; begins honoring February 1, 2027	Recognized as a training opt-out during web crawling	Decoupled from training; managed separately via page-level tags	Retires NoArchive/NoCache for AI on March 1, 2027; temporary Bing Webmaster Tools setting opens Dec 15
OpenAI	Committed to implementation in upcoming months	Direct support in data ingestion pipelines	Evaluated alongside GPTBot directives	Existing GPTBot configurations remain active and enforceable
Anthropic	Employee authored draft; development reported on working group list	Intended support across services; formal newsroom timeline TBD	Evaluated alongside ClaudeBot directives	Existing ClaudeBot crawler controls continue to operate

Google plans to interpret a shared AI training opt-out as a Google-Extended disallow when no explicit Google-Extended rule is present. Because Google-Extended also governs eligibility for grounding in Gemini Apps, this default mapping can affect whether a publisher’s content is available for those experiences. An explicit Google-Extended rule takes precedence over the shared signal in Google’s implementation, but publishers should not assume that Google’s current controls offer independent opt-outs for model training and Gemini Apps grounding. Content permitted under Google-Extended may be eligible for Gemini grounding, without any guarantee of inclusion or citation.

In contrast, Microsoft Bing treats model training and real-time grounding as distinct architectural concepts, as detailed in the official Bing Webmaster update. Microsoft allows publishers to opt out of generative model training while continuing to permit live grounding in Copilot answers. Sites managing these preferences can declare training restrictions in robots.txt or via page-level HTML meta tags and HTTP headers, which override robots.txt for individual URLs. Grounding is controlled via page-level preferences, including a DisallowExtended option that permits grounding in Microsoft’s own tools (like Copilot) while excluding third-party services powered by Microsoft grounding APIs. Furthermore, Microsoft is phasing out its historical reliance on NoArchive and NoCache tags for AI controls; sites relying on those tags must migrate to the new options before March 1, 2027. The temporary Bing Webmaster Tools grounding setting is available only to sites using NoArchive or NoCache as of December 15, 2026. It remains effective until Microsoft recrawls the page and discovers the new preference, or until September 30, 2027, whichever occurs first.

Engineering Checklist: Auditing Web Preferences for the Agentic Web

Deploying the shared AI training preference signal requires coordination between systems architecture, content operations, and legal governance teams. Engineering leaders must ensure that server configurations accurately reflect organizational policies without inadvertently disrupting search engine discovery.

Webmaster and Systems Implementation Checklist

• Audit Current Robots Manifests: Review existing robots.txt files to catalog vendor-specific directives, verifying whether rules for GPTBot, ClaudeBot, or Google-Extended conflict with organizational training policies.
• Deploy the Reserved Policy Token: Implement the User-Agent: __usage_ai-training block to express global training boundaries across participating AI providers in a unified format.
• Maintain Granular Exception Paths: If public assets—such as press releases, media kits, or open developer documentation—are intended for model consumption, declare explicit Allow: exceptions below the disallow directive.
• Separate Fetch Blocking from Training Governance: If the primary objective is reducing origin server load or preventing unauthorized scraping, configure IP-based rate limiting, CDN bot challenges, and explicit crawler user-agent blocks.

Governance and Compliance Checklist

• Review Microsoft Grounding Migration Schedules: For sites utilizing NoArchive or NoCache to control Bing Copilot output, prepare to transition to the new page-level tags prior to Microsoft’s March 1, 2027 enforcement deadline.
• Assess Search Discovery Trade-offs: For platforms evaluating visibility in conversational answer engines, recognize how vendor-specific default mappings (such as Google’s fallback to Google-Extended) affect real-time retrieval eligibility.
• Track IETF Working Group Updates: Monitor the progression of draft-kelley-aipref-training-preference-signal within the IETF AIPREF working group to account for potential syntax updates before its scheduled expiration in May 2027.