Robots.txt is not the answer: Proposing a new meta tag for LLM/AI

While Google is opening up the discussion on giving credit and adhering to copyright when training large language models (LLMs) for generative AI products, their focus is on the robots.txt file.

However, in my opinion, this is the wrong tool to look at.

My former colleague Pierre Far wrote an excellent article on Crawlers, search engines and the sleaze of generative AI companies where he highlighted some of the immense challenges currently facing the online publishing industry. Similar to his article, I will keep this proposal high-level as developments in this field are extremely fast-paced.

Why not use robots.txt

There are a few reasons why using robots.txt is the wrong starting point for the discussion on how to respect the copyright of publishers.

Not all LLMs use crawlers and identify themselves

The burden is on the website operator to identify and block individual crawlers, which may use and/or sell their data for generative AI products. This creates a lot of extra (and unnecessary) work, especially for smaller publishers.

This also assumes that the publisher has editing access to their robots.txt file, which is not always the case with hosted solutions.

This is not a sustainable solution as the number of crawlers continues to grow

The usable file size of a robots.txt file is limited to 500 kb, according to the newly proposed robots.txt standard.

This means that a large publisher may run into problems with their robots.txt file if they need to block a lot of LLM crawlers and/or refined URL patterns in addition to other bots.

An ‘all or nothing’ approach is unacceptable

For the larger crawlers like Googlebot and Bingbot, no distinction can be made between the data being used for search engine results pages (traditionally where there is an “agreement“ between the publisher and search engine in the shape of a “citation“ to the original source) and generative AI products.

Blocking Googlebot or Bingbot for their generative AI products also blocks any potential visibility in their respective search results. This is an unacceptable situation where the publisher is forced to make a choice between “all or nothing”.

Robots.txt is all about managing crawling while the copyright discussion is all about how the data is used

The latter is about the indexation/processing phase. As such, robots.txt isn’t really relevant to this discussion but rather a last resort if nothing else works and should really not be the starting point of this particular discussion.

Robots.txt files work fine for crawlers and do not need changing for the purpose of LLMs. Yes, LLM crawlers need to identify themselves, but what we really need to talk about is indexation/processing of the crawled data.

Reinventing the wheel

Luckily, the web already has some well-established solutions which can be used for managing the usage of data with regard to copyrights. It is called Creative Commons.

Most of the Creative Commons licenses would do fine for the purpose of LLMs. To illustrate:

• CC0 allows LLMs to distribute, remix, adapt, and build upon the material in any medium or format with no conditions.
• CC BY allows LLMs to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use, but credit must be given to the creator.
• CC BY-SA allows LLMs to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use. If LLMs remix, adapt, or build upon the material, it must license the modified material under identical terms.
• CC BY-NC allows LLMs to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only so long as attribution is given to the creator. 
• CC BY-NC-SA allows LLMs to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only so long as attribution is given to the creator. If LLMs remix, adapt, or build upon the material, they must license the modified material under identical terms. 
• CC BY-ND allows LLMs to copy and distribute the material in any medium or format in unadapted form only so long as attribution is given to the creator. The license allows for commercial use and credit must be given to the creator, but no derivatives or adaptations of the work are permitted. 
• CC BY-NC-ND allows LLMs to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and so long as attribution is given to the creator and no derivatives or adaptations of the work are permitted. 

The last two licenses are unlikely to be usable for LLMs.

However, the first five licenses mean that LLMs need to consider how they use the crawled/obtained data and ensure they adhere to the requirements placed upon using the data from the publishers, such as attribution and when sharing the product built upon the data.

This would put the burden on the “few“ LLMs in the world instead of the “many“ publishers.

The first three licenses also support “traditional“ usage of the data, for example, in search engine results where the attribution/credit is given through the link to the original website. While the fourth and the fifth license also support research and development for open-source LLMs. 

Side note: Keep in mind that all these software companies building LLMs often use open-source software where they have the same copyright license challenges with regard to the software libraries and operating systems they use to avoid copyright violations on a code level. So why reinvent the wheel when we can use a similar system for the data this code processes?

The meta tag is the way

Once a publisher has identified an appropriate license, this license still needs to be communicated. Again, this is where robots.txt seems to be the wrong approach.

Just because a page should be blocked from crawling for search engines does not mean it can’t be used or isn’t useful for LLMs. These are two different use cases.

As such, to separate these use cases and allow for a more refined yet also easier approach for publishers, I recommend we use a meta tag instead.

Meta tags are pieces of code that can be inserted on a page level, within a theme or the content (I know, this is not technically correct, but HTML is forgiving enough and can be used as a last resort when a publisher has limited access to the code base). They do not require the publisher to have additional access rights other than being able to edit the HTML of the content published.

Using meta tags does not stop crawling, like the meta noindex. However, it allows you to communicate the usage rights of the data published.

And although there are existing copyright tags that can be used – notably from Dublin Core, rights-standard (abandoned proposal), copyright-meta (focuses on the name of the owner rather than the license) and other attempts – the current existing implementation of these on some websites may conflict with what we try to accomplish here.

So a new meta tag may be necessary, although I am happy to reuse an existing or old one, such as “rights-standard“, as well. For this discussion, I am proposing the following new meta tag:

In addition, I recommend that this meta tag is also supported when used in HTTP Headers, like the noindex is supported in X-Robots-Tag, to aid LLMs crawlers better managing their crawl resources (they only need to check the HTTP Headers to validate the usage rights).

X-Robots-Tag: usage-rights: CC-BY-SA

This can be used in combination with other meta tags. In the example below, the page should not be used for search results but can be used for commercial LLMs as long credit is given to the source:

X-Robots-Tag: usage-rights: CC-BY, noindex

Note: The name “usage-rights“ for the meta tag is a proposal and can be changed.

Foolproof solution

Granted, there are bad crawlers and bad actors building their LLMs and generative AI products.

The proposed meta tag solution won’t prevent the content from being used that way, but neither will the robots.txt file.

It is important to acknowledge that both methods depend on the recognition and compliance by the companies using the data for their AI products.

Conclusion

Hopefully, this article illustrates how using robots.txt for managing data usage in LLMs is, in my opinion, the wrong approach/starting point for dealing with usage and copyrights in this new age of LLMs and generative AI products.

This meta tag implementation would enable publishers to specify copyright information at the page level using Creative Commons, without preventing the page from being crawled or indexed for other purposes (like search engine results). It also allows for copyright declarations to be made for various uses, including LLMs, generative AI products, and potential future AI products.

The post Robots.txt is not the answer: Proposing a new meta tag for LLM/AI appeared first on Search Engine Land.