Brozzler

Brozzler is a distributed web crawler that drives a real browser (Chrome or Chromium) to fetch pages, extract embedded links, and capture media via yt-dlp,…

  • Backup
  • Development
  • Tools & Utilities
Brozzler is a distributed web crawler that drives a real browser (Chrome or Chromium) to fetch pages, extract embedded links, and capture media via yt-dlp, purpose-built to pair with WARC-based archiving tools like WARCprox for large web-archiving projects.

Platform availability

Available on 0 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

53/100 Fair

Maintained
88last commit 2 days ago
Popular
58824 GitHub stars
Easy to install
250 install platforms, install notes, Docker image
Light to run
No data
Security check
54 serious findings, 14 minor, CI, tests, OpenSSF Scorecard 3.6/10

Based on 4 of 5 factors.

GitHub stars
824
Open issues
63
Last commit
2026-11-19
Latest release
v1.10.0
License
Apache-2.0
Activity
rising

Checked 2 days ago - source: GitHub (internetarchive/brozzler)

Resources & Compatibility

ARM (e.g. Raspberry Pi) is not supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Start RethinkDB before jobs Install Brozzler with the RethinkDB extra and start the RethinkDB (a database service Brozzler uses) before running `brozzler-new-job` or `brozzler-worker`. Brozzler uses it to track jobs; if it is not running, jobs may fail to start. source
  • Create a simple job file Make a YAML job file with an id and at least one starting URL under `seeds`, and if using Vagrant set `proxy: localhost:8000`. Then run `brozzler-new-job job1.yml`; without a seed URL, the crawler has nowhere to begin. source
  • Use scope to keep it on target Add `scope` rules such as accepts, blocks, and `max_hops` so the crawler only visits pages you intended. This avoids wasting time on unrelated sites, login pages, or other pages you do not want archived. source
  • Provide a display for the browser Brozzler drives a real browser, which needs a graphical environment to run; on a headless server you must supply one (e.g. a virtual display), as stated in the README. source

Fetched 42 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

Which X11 variant is recommended when running Brozzler on a server?

Xvnc4 is preferred over Xvfb for server use because Chromium on Xvfb did not support screenshots when last tested.

How does Brozzler find a warcprox instance in a clustered deployment?

Run workers with --warcprox-auto so Brozzler looks for a healthy warcprox instance in the RethinkDB service registry. The warcprox instance must point to the same RethinkDB services table.

What installation issue is reported for Brozzler [easy] on Cygwin and Linux?

The issue reports that Brozzler 1.5.18 installs, but Brozzler [easy] does not due to various dependency conflicts.

How can I quickly test Brozzler on one URL?

Use the brozzle-page command-line utility and pass the URL to crawl.

Answers sourced from github.com, github.com

Community

Collected 48 days ago - public community threads