Brands back new licensing standard to charge AI for training data

Major publishers – including Reddit, Yahoo, Medium, and Quora – today announced support of a new licensing standard, Really Simple Licensing (RSL). In theory, RSL gives websites leverage to demand compensation when AI companies scrape and train on their data.

Why we care. Until now, publishers could only block or allow bots via robots.txt. RSL upgrades that model, letting publishers embed licensing and royalty terms directly into robots.txt files, online books, videos, and datasets.

How it works. Publishers can set subscription, pay-per-crawl, or pay-per-inference fees (compensation when an AI uses their content in responses).

• Bots for non-training purposes (like search indexing or archiving) can crawl as usual.
• RSL relies on AI companies’ cooperation. Without buy-in, enforcement will be a challenge.

The big picture. The RSL Collective, led by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, is pushing the standard.

• They see it as a scalable business model for the web, modeled after rights groups like ASCAP in music.
• Fastly is working with RSL to build “gatekeeper” tech that can admit or block bots based on licensing compliance.

Supporters. Brands that are supporting RSL include:

• Reddit
• Quora
• Yahoo
• Medium
• Adweek
• Fastly
• Internet Brands
• The MIT Press
• O’Reilly
• People Inc.
• wikiHow
• Ziff Davis

Between the lines. Some big media players (e.g., The New York Times, News Corp, Vox Media) have cut direct licensing deals with AI firms. But RSL aims to simplify the process for any publisher, big or small.

The catch. AI model builders have a history of ignoring robots.txt. Without enforcement teeth, RSL’s success hinges on whether major AI players adopt the standard.

Not the first. Cloudflare, which powers about 20% of the internet, is also cracking down on AI bots. It blocks AI crawlers by default and is testing a Pay Per Crawl system that charges AI companies for access – a parallel effort to RSL that could reshape how AI firms get training data.

Really Simple Licensing. You can dig deeper on the RSL website.