← Directory

Crawl4AI

Open-source web crawler and scraper tailored for LLMs, AI agents, and data pipelines.

  • AI
  • Other
  • Productivity
  • Tools & Utilities
Open-source web crawler and scraper tailored for LLMs, AI agents, and data pipelines.

Platform availability

Available on 1 of 9 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

85/100 Excellent

Maintained
100last commit 8 days ago
Popular
9985,009 GitHub stars
Easy to install
351 install platform, install notes, Docker image
Light to run
No data
Security check
88no serious findings, 2 minor, CI, tests

Based on 4 of 5 factors.

Availability: platform listings confirmed today, links checked 13 days ago

GitHub stars
85,009
Open issues
232
Last commit
2026-10-05
Latest release
v0.9.4
License
Apache-2.0
Activity
steady

Checked 5 days ago - source: GitHub (unclecode/crawl4ai)

Resources & Compatibility

ARM (e.g. Raspberry Pi) is supported.

Checked 14 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Add AI keys before running self-hosted If you are running Crawl4AI yourself, copy deploy/docker/.llm.env.example to .llm.env in the main Crawl4AI folder and add your LLM (large language model) API keys. Without this file, AI-based features may not work correctly. source
  • Test your install with a hello-world example After installing Crawl4AI, run the simple hello-world example from the docs to confirm everything is set up. If this first test fails, fix the installation before trying more advanced examples. source
  • Docker examples need a local server The Docker client example connects to http://localhost:11235, so make sure that Crawl4AI server is running and reachable first. If the address or port is wrong, the example will report a crawl failure. source

Fetched 14 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How do I enable AI-based features in a self-hosted Crawl4AI setup?

Copy deploy/docker/.llm.env.example to .llm.env in the main Crawl4AI folder and add your LLM API keys. Without this file, AI-based features may not work correctly.

How is the browser installed when using Crawl4AI from the Python package?

After installing crawl4ai, run crawl4ai-setup, which installs the browser once.

What address do the Docker examples expect the self-hosted Crawl4AI server to use?

They expect the server on localhost at port 11235. If the address or port is wrong, the example reports a crawl failure.

How can I confirm a fresh Crawl4AI installation works?

Run the simple hello-world example from the docs after installing. If it fails, fix the installation before trying advanced examples.

Does Crawl4AI reliably bypass strict anti-crawling protections?

It can simulate users to try to avoid risks, but websites with strict anti-crawling can still be difficult.

Do I need a Crawl4AI Cloud key to self-host?

No, self-hosting is a separate option: you can run it yourself as a library or as your own Docker server. Crawl4AI Cloud is the hosted version.

Answers sourced from docs.crawl4ai.com, docs.crawl4ai.com, github.com, github.com

Community

Collected 13 days ago - public community threads