The open source monthly Web crawl Common Crawl now “contains 3.08 billion+ web pages” and 250TB of uncompressed data. Back in September 2015, when this blog last took a look at Common Crawl, that was 145TB.
Common Crawl at 3bn Web pages
11 Saturday Mar 2017
Posted in Spotted in the news