Major AI labs adopt shared IETF standard for scraping opt-outs

OpenAI, Google, Microsoft, and Anthropic have agreed to support a new Internet Engineering Task Force (IETF) standard that allows website publishers to opt out of AI data scraping. Announced on October 30, 2026, this initiative introduces a shared policy token within the existing robots.txt framework, enabling site owners to set training preferences once rather than negotiating with each AI provider individually.

What happened

The core of this development is a unified signal proposed by Kevin Kelley, a technical lead for standards at Anthropic. Currently, publishers who wish to prevent their content from being used in AI model training must configure separate settings for every major AI company. This fragmented approach creates significant administrative overhead and often leads to inconsistent enforcement. The new standard aims to consolidate these preferences into a single, machine-readable directive that all participating crawlers will respect.

Under this agreement, if an AI crawler encounters the shared policy token in a site’s robots.txt file, it will restrict access to that site for training purposes. Anthropic has already adopted the signal. OpenAI plans to implement it in the coming months, while Google and Microsoft have scheduled support for next year. Meta has also expressed support for the concept, with communications head Andy Stone noting the importance of a common signal, though the company has not yet confirmed whether it will technically honor the opt-out.

This move arrives as publishers continue to struggle with the volume of automated traffic generated by AI bots. Previous attempts to manage this influx include the Really Simple Licensing (RSL) standard, which allows publishers to charge for crawler access, and infrastructure-level blocks like those offered by Cloudflare. The new IETF proposal differs by focusing on a standardized, voluntary compliance mechanism among the largest AI developers rather than payment gateways or hard network blocks.

How it works

The technology leverages the robots.txt file, a long-standing text file placed in the root directory of websites to communicate with web crawlers. Traditionally, this file tells search engines which pages they can index for search results. The new standard extends this protocol by introducing a specific policy token that signals preferences regarding AI training and grounding.

When an AI company’s crawler visits a website, it first checks the robots.txt file. If it finds the new policy token indicating an opt-out, the crawler is programmed to skip the site’s content for the specified purposes. Microsoft has indicated it will offer granular controls, allowing publishers to distinguish between opting out of general AI training and opting out of AI grounding. Grounding refers to the process where models reference verified sources to reduce hallucinations during generation. Importantly, Microsoft clarified that opting out of AI training or grounding will not affect a site’s visibility in Bing’s traditional search results.

Google has implemented similar distinctions in the past, allowing sites to control their presence in AI-generated search features without penalizing their organic search ranking. The new IETF standard seeks to make this level of control interoperable across different providers, ensuring that a single configuration update propagates to all compliant crawlers.

Key details

• Proposer: Kevin Kelley, technical lead for standards at Anthropic, submitted the proposal to the IETF.
• Mechanism: A shared policy token added to the website’s robots.txt file.
• Adoption timeline: Anthropic has already adopted the signal; OpenAI will implement it in the coming months; Google and Microsoft plan to support it next year.
• Meta’s stance: Andy Stone expressed support for the standard but did not confirm if Meta will honor the opt-out technically.
• Granularity: Microsoft will allow publishers to separately opt out of AI training and AI grounding.
• Search impact: Opting out of AI scraping will not remove websites from traditional search engine results like Bing or Google Search.

Why it matters

For software engineers and technical founders building products that rely on web data, this standardization simplifies the compliance landscape. Instead of maintaining complex, provider-specific allowlists or blocklists, engineering teams can rely on a single, industry-backed signal to determine data usage rights. This reduces the risk of accidental copyright infringement and lowers the operational cost of managing data ingestion pipelines.

However, the staggered rollout means that immediate uniformity is not guaranteed. Developers building RAG (Retrieval-Augmented Generation) systems or training foundational models must remain aware that not all major players are enforcing the token simultaneously. Until Google and Microsoft fully implement the standard next year, publishers may still need to maintain legacy configurations for those specific crawlers. This transitional period requires careful monitoring of crawler behavior and adherence to existing individual policies until the ecosystem converges.

Furthermore, the distinction between training and grounding is critical for application architecture. If a publisher opts out of grounding but allows training, their content might influence the model’s weights but not be cited as a source in real-time responses. Engineers must design their data filtering logic to respect these nuanced preferences, ensuring that their applications do not inadvertently violate a publisher’s specific constraints regarding how their data is utilized.

What you can do

• Audit your current data ingestion pipelines to identify which AI crawlers are currently accessing your domains.
• Update your robots.txt files to include the new policy token if you wish to opt out of AI training, keeping in mind that full industry support is not yet universal.
• Monitor the implementation timelines for Google and Microsoft, as their support is scheduled for next year, requiring continued use of their specific legacy controls in the interim.
• Review your RAG system’s sourcing logic to ensure it respects granular opt-outs for AI grounding, particularly if you use Microsoft-based models.
• Stay informed about Meta’s final decision on honoring the standard, as their participation remains unconfirmed despite public support.
• Document your compliance strategy internally to ensure that data procurement practices align with the evolving IETF standard and individual provider policies.