aiqbench takeAI Policy

Four major AI developers have committed to backing a single robots.txt entry that would let site operators block generative AI training scrapers, replacing the current per-provider, per-scraper setup. The standard is still a draft submitted to the IETF, so its effect depends on adoption and on whether compliance is voluntary. Readers should watch how it defines "generative" versus "exclusively non-generative" uses, since that boundary determines what publishers can actually control.

Sources coveringSiliconANGLE, AWS

Google, Anthropic and others to support new AI crawling opt-out standard

SiliconANGLEOctober 31, 20263 min read

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […] The post Google, Anthropic and others to support new AI crawling opt-out standard appeared first on SiliconANGLE.

Excerpt shown under the publisher's RSS terms · 56 words · full text and figures are on the publisher's site

Read at SiliconANGLE ↗Ask about this article