Consent In Crisis The Rapid Decline Of The Ai Data Commons 2024 07 19

A publicly accessible webpage does not necessarily reflect its publisher's willingness to supply AI training data. The Consent in Crisis abstract, hosted by Cohere, reports an audit of 14,000 web domains and a sharp rise in restrictions during 2023–2024. Researchers found discrepancies between restrictions expressed in website terms and those expressed through crawler instructions. They argue that honoring the growing restrictions could change the freshness and diversity of training datasets. The finding concerns publishers' changing stated preferences and their uneven expression across web protocols.