Common crawl’s massive internet archive may be giving ai companies access to paywalled journalism, according to a new report. A nonprofit organization has been systematically supplying paywalled news articles to major ai companies for training large language models, according to an investigation published november 4, 2025, by the atlantic's alex reisner Common crawl maintains archives containing millions of articles from major news organizations that readers typically must pay to access, enabling ai developers. The company quietly funneling paywalled articles to ai developers the atlantic / alex reisner / nov 5, 2025 “a search for nytimes.com in any crawl from 2013 through 2022 shows a ‘no captures’ result, when in fact there are articles from nytimes.com in most of these crawls. The atlantic on common crawl, the nonprofit funneling paywalled articles to ai companies a brutally efficient exposé, alex reisner caught them in several lies by simply looking at their crawl data (via) Nonprofit common crawl, best known for making open web archives that are used to train ai models, is under fire for claims that its troves have allowed ai companies to ingest paywalled journalism at scale
The organization’s data in turn has acted as a kind of workaround for model creators looking to add premium news articles (from the new york times, the washington post, and wired, among.
WATCH
