Crawl Log Viewer

Crawl Log Viewer is a standalone web service that retrieves and filters crawl log streams stored as Apache Kafka topics, showing crawler activity with…

  • Development
  • Monitoring & Logging
Crawl Log Viewer is a standalone web service that retrieves and filters crawl log streams stored as Apache Kafka topics, showing crawler activity with hoverable HTTP status-code explanations so operators running large-scale crawls can quickly verify what their bots collected.

Platform availability

Available on 0 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

12/100 Low

Maintained
0last commit 1026 days ago
Popular
00 GitHub stars
Easy to install
250 install platforms, install notes, Docker image
Light to run
502 GB minimum RAM
GitHub stars
0
Open issues
0
Last commit
2024-01-30
License
Apache-2.0
Activity
stale

Checked 18 days ago - source: GitHub (ldbiz/crawl-log-viewer)

Resources & Compatibility

Minimum requirements: 2 GB RAM, 2 CPU core(s). ARM (e.g. Raspberry Pi) is not supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Install the Python pieces it needs Make sure Python and pip (the Python package installer) are available, then run `python -m pip install -r requirements.txt`. Without these packages, the app’s commands can fail because they can’t import the code they need. source
  • Fill in the MCP client settings In your MCP (Model Context Protocol) client config, set the Python executable, the path to `server.py`, and `SEARXNG_URL` if you’re using SearXNG (a search engine). If these paths or URLs are wrong, the client can’t start the tool or won’t know where search requests should go. source
  • Start it from the right folder After setup, start the server from the crawl folder with `crawl-mcp` or `python server.py`. If you use the wrong folder, the MCP client may not be able to reach the server. source
  • Configure topics.json for your Kafka topics Create a topics.json configuration file pointing the service at your Kafka brokers and the crawl log topics to inspect; the view defaults to the previous day's activity from the first topic in the list. source

Fetched 43 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

Can a crawl log be used to diagnose crawler problems?

Yes. A crawl log can show the status of crawled content and help diagnose problems.

Can it work alongside existing self-hosted monitoring tools?

A self-hoster asked for a lightweight, Docker-friendly web log viewer to complement tools such as Netdata, Dozzle, and Glances.

Can it view non-Docker log files without a full log aggregator?

A self-hoster asked for a Dozzle-like clean web interface for non-Docker log files, without using a full log aggregator like Loki.

What do self-hosters want in a home server log monitoring setup?

A self-hoster wanted a simple, maintainable log monitoring solution for a home server.

Answers sourced from support.microsoft.com, www.reddit.com, www.reddit.com, www.reddit.com

Community

Collected 48 days ago - public community threads