In a detailed investigation for the atlantic, reporter alex reisner reveals that several major ai companies have quietly partnered with the common crawl foundation — a nonprofit that scrapes. Multiple news publishers requested that common crawl remove their articles to prevent ai training use Common crawl claims it complies with these requests, but research shows otherwise. The company quietly funneling paywalled articles to ai developers the atlantic / alex reisner / nov 5, 2025 “a search for nytimes.com in any crawl from 2013 through 2022 shows a ‘no captures’ result, when in fact there are articles from nytimes.com in most of these crawls. The common crawl foundation has been scraping the internet for over a decade, creating a vast archive used by ai companies to train models, including paywalled content Despite claims of compliance with publishers' requests to remove their articles, investigations reveal that many remain in the archive
The foundation's director argues for the right of ai to access all internet content. The atlantic on common crawl, the nonprofit funneling paywalled articles to ai companies a brutally efficient exposé, alex reisner caught them in several lies by simply looking at their crawl data (via)
WATCH
