In the process, my reporting has found, common crawl has opened a back door for ai companies to train their models with paywalled articles from major news websites. According to the report, common crawl, whose database spans multiple petabytes, has effectively opened a backdoor that allows ai companies to train their models on paywalled content from. Common crawl maintains archives containing millions of articles from major news organizations that readers typically must pay to access, enabling ai developers including openai, google, anthropic, nvidia, meta, and amazon to train their models on premium journalism without compensation to publishers. Nonprofit common crawl, best known for making open web archives that are used to train ai models, is under fire for claims that its troves have allowed ai companies to ingest paywalled journalism at scale. In the evolving landscape of artificial intelligence (ai), a quiet controversy has emerged surrounding the common crawl foundation, a nonprofit organization that has been collecting data from billions of webpages for over a decade.
WATCH
