Common Crawl has updated and is now at “3.2 billion Web pages and 260 TiB of uncompressed content”. In September they added a list of university domains to the crawl. This time, for the first time, they’ve actively been trying to blacklist spam-network pages.

The Crawl is also now including 300m+ new URLs from a paid service called Mixnode, which looks like a very interesting on-demand custom-crawling and indexing service…

“Mixnode can breeze through thousands of URLs per second and download gigabytes of data per minute without a hitch.”

Presumably some of this comes via abstracting sections of the Common Crawl, then ‘filling in’ the rest?

Now all Mixnode needs is a half-decent ‘public search’ front-end for a Mixnode crawl, and it’s ‘Build Your Own Search Engine’ time — without the limitations of a Google CSE.

Nothing about this on Google News, but apparently the universities-only oaFindr has dropped the OA bit of the name and is now called 1Findr…

“1science has decided to change the name of the product line. Market analysis shows that its greatest value is the analysis and the integral access to the content, and to reflect this change, they have changed the prefix “oa” to “1”. So, oaFindr becomes 1Findr; oaFigr becomes 1Figr; and oaFoldr + becomes 1Foldr Data.”

Por un lado, 1science ha decidido cambiar el nombre de línea de productos. Puesto que su mayor valor es el análisis y el acceso integral al contenido, para reflejar este cambio, han cambiado el prefijo “oa” por “1”. Así pues, oaFindr se convierte en 1findr; oaFigr se convierte en 1figr; y oaFoldr + se convierte en 1foldr Data. … proporciona hipervínculos a 30 millones de artículos disponibles en acceso abierto.

Source, SEDEC Bulletin, Nov 2017.

The same source also quotes some current statistics. According to the CEO 1Findr currently … “provides hyperlinks to 30 million articles available in Open Access.” as part of a wider mixed database of records for 85 million. That’s both paywalled and OA academic articles. Apparently dating from the 17th century onwards, interestingly. In comparison, Microsoft Academic’s last known total (late 2016) was 140 million.