Crawl4AI

Open-source, self-hosted web crawler and scraper that converts web pages into clean Markdown and structured JSON, built for AI agents and LLM data pipelines.

  • AI
  • Other
  • Productivity
  • Tools & Utilities
Open-source, self-hosted web crawler and scraper that converts web pages into clean Markdown and structured JSON, built for AI agents and LLM data pipelines.

Platform availability

Available on 2 of 10 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

84/100 Excellent

Maintained
100last commit 26 days ago
Popular
9986,712 GitHub stars
Easy to install
452 install platforms, install notes, Docker image
Light to run
602 GB minimum RAM, runs on ARM
Security check
88no serious findings, 2 minor, CI, tests

Availability: platform listings confirmed today, links checked 53 days ago

GitHub stars
86,712
Open issues
308
Last commit
2026-10-26
Latest release
v0.9.5
License
Apache-2.0
Activity
declining

Checked 20 days ago - source: GitHub (unclecode/crawl4ai)

Resources & Compatibility

Minimum requirements: 2 GB RAM, 2 GB disk. ARM (e.g. Raspberry Pi) is supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Add AI keys before running self-hosted If you are running Crawl4AI yourself, copy deploy/docker/.llm.env.example to .llm.env in the main Crawl4AI folder and add your LLM (large language model) API keys. Without this file, AI-based features may not work correctly. source
  • Test your install with a hello-world example After installing Crawl4AI, run the simple hello-world example from the docs to confirm everything is set up. If this first test fails, fix the installation before trying more advanced examples. source
  • Docker examples need a local server The Docker client example connects to http://localhost:11235, so make sure that Crawl4AI server is running and reachable first. If the address or port is wrong, the example will report a crawl failure. source

Fetched 54 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How do I enable AI-based features in a self-hosted Crawl4AI setup?

Copy deploy/docker/.llm.env.example to .llm.env in the main Crawl4AI folder and add your LLM API keys. Without this file, AI-based features may not work correctly.

How is the browser installed when using Crawl4AI from the Python package?

After installing crawl4ai, run crawl4ai-setup, which installs the browser once.

What address do the Docker examples expect the self-hosted Crawl4AI server to use?

They expect the server on localhost at port 11235. If the address or port is wrong, the example reports a crawl failure.

How can I confirm a fresh Crawl4AI installation works?

Run the simple hello-world example from the docs after installing. If it fails, fix the installation before trying advanced examples.

Does Crawl4AI reliably bypass strict anti-crawling protections?

It can simulate users to try to avoid risks, but websites with strict anti-crawling can still be difficult.

Do I need a Crawl4AI Cloud key to self-host?

No, self-hosting is a separate option: you can run it yourself as a library or as your own Docker server. Crawl4AI Cloud is the hosted version.

Answers sourced from docs.crawl4ai.com, docs.crawl4ai.com, github.com, github.com

Community

Collected 53 days ago - public community threads