Nutch

Apache Nutch is a highly extensible and scalable open source web crawler for developers who build large-scale search indexes and extract content from websites.

  • Development
  • Tools & Utilities
  • Web & Publishing
Apache Nutch is a highly extensible and scalable open source web crawler for developers who build large-scale search indexes and extract content from websites.

Platform availability

Available on 0 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

63/100 Good

Maintained
100last commit 10 days ago
Popular
703,312 GitHub stars
Easy to install
250 install platforms, install notes, Docker image
Light to run
No data
Security check
201 serious finding, CI, tests, OpenSSF Scorecard 5.4/10

Based on 4 of 5 factors.

GitHub stars
3,312
Open issues
7
Last commit
2026-11-11
License
Apache-2.0
Activity
steady

Checked 3 days ago - source: GitHub (apache/nutch)

Resources & Compatibility

ARM (e.g. Raspberry Pi) is not supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Start Docker before running Nutch If you are using the Docker setup, start Docker and load the environment variables before launching Nutch containers. This makes Docker commands available and avoids failures when starting the app. source
  • Install Firefox for local Selenium If Nutch is set to use Selenium in local mode with Firefox, install Firefox before crawling. Without the browser, Selenium cannot open pages and the web crawl will fail. source
  • Install and start Xvfb first On Ubuntu, install Xvfb (a virtual display service) and its helper packages, then start that display before you start a web crawl. It gives Firefox a screen to run on without a monitor, so missing it can stop local crawling. source

Fetched 53 days ago - each note links to its source

Alternatives (3)

  • Scraperr - Scraperr is a self-hosted, no-code web scraping tool for homelab users who want to extract structured page data without coding, using XPath…
  • Scrapy - Scrapy is a free and open-source web-crawling framework written in Python.
  • StormCrawler - StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.

Checked 54 days ago - source: AlternativeTo

Common questions

What should I do if the Docker setup fails to start Nutch containers?

Start Docker and load the environment variables before launching Nutch containers. This makes Docker commands available and avoids failures when starting the app.

Why does Nutch say no plugins were found?

Check that nutch-site.xml is configured and that plugin.folders points to the plugin directory, normally /build/plugins. Changing plugin.folders in nutch-default.xml is a quick fix but should not be used.

Answers sourced from github.com, github.com

Community

Collected 48 days ago - public community threads