Common crawl’s massive internet archive may be giving ai companies access to paywalled journalism, according to a new report. The company quietly funneling paywalled articles to ai developers the atlantic / alex reisner / nov 5, 2025 “a search for nytimes.com in any crawl from 2013 through 2022 shows a ‘no captures’ result, when in fact there are articles from nytimes.com in most of these crawls. A nonprofit organization has been systematically supplying paywalled news articles to major ai companies for training large language models, according to an investigation published november 4, 2025, by the atlantic's alex reisner Common crawl maintains archives containing millions of articles from major news organizations that readers typically must pay to access, enabling ai developers. At the request of brein, common crawl has removed over two million news articles belonging to popular dutch news outlets from its ai training dataset The common crawl foundation has been scraping the internet for over a decade, creating a vast archive used by ai companies to train models, including paywalled content.
WATCH
