Law firms and other publishers can now block OpenAI, Google, Microsoft, and Anthropic from training on their website content with a single robots.txt policy token, instead of configuring each vendor separately. For firms worried about client-adjacent material or jurisdiction-sensitive commentary being absorbed into AI models, this is a one-line governance action that applies across all four major providers at once.
According to The Verge, the shared signal was proposed through the IETF by an Anthropic standards lead and lives inside the robots.txt file. Anthropic already respects it; Google and Microsoft plan to in 2027; OpenAI says "coming months". Meta signalled support but has not committed to enforce it. Microsoft adds granular controls: opt out of training separately from AI grounding, while keeping Bing search visibility.
What this changes for a small business
If you publish anything public - a knowledge base, product catalog, legal explainers, case studies - and you do not want it feeding future models, you edit one file instead of filling four vendor forms. For a law firm running an internal legal document review pipeline, this closes one leak at the perimeter: public commentary is governed, and private contract archives stay inside your own contour where they belonged in the first place. Note the limit - the signal is voluntary; smaller or non-signatory crawlers will still ignore it, so Cloudflare-style blocking remains relevant.
Why it matters: Public-content governance across the four biggest AI labs just became a five-minute robots.txt edit - add it to your compliance checklist this week.
Source: The Verge