Apache Nutch

Apache Nutch is a mature, production-grade web crawler for collecting data from the web at scale. Written in Java, it runs crawl jobs on Apache Hadoop and integrates with search indexes such as Apache Solr.

Crawling is organized into configurable workflows rather than a fixed model, and a plugin architecture lets you adapt fetching, parsing, indexing, and scoring to different acquisition needs. This makes it suitable for everything from focused crawls to broad, distributed collection.

Nutch is an Apache Software Foundation project, released under the Apache License 2.0. It is self-hosted, with no hosted service, and is governed through Apache's community process.