Experiences with the Nutch search engine

Nutch is open-source software that implements a web search engine. It has been used in a variety of applications: vertical search engines, archival web search, search engines that incorporate novel metadata, etc. Nutch is itself implemented using Hadoop, an open-source platform for scalable computing. Hadoop facilitates the development and management of applications that run on large numbers of computers and on very large datasets. Hadoop has been demonstrated on clusters with hundreds of computers and is designed to scale to thousands of computers. This talk will present the architecture, capabilities and current status of these two projects.