Beam Spark Job Server
Containerized Apache Beam Spark job server that executes batch and streaming data-parallel pipelines on an Apache Spark cluster, letting self-hosters run Beam…
Platform availability
Available on 0 of 10 platforms
- Unraid (not listed)
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
66/100 Good
- Maintained
- 100last commit 16 days ago
- Popular
- 798,709 GitHub stars
- Easy to install
- 250 install platforms, install notes, Docker image
- Light to run
- 108 GB minimum RAM
- GitHub stars
- 8,709
- Open issues
- 3868
- Last commit
- 2026-11-05
- Latest release
- v2.77.0
- License
- Apache-2.0
- Activity
- rising
Checked 16 days ago - source: GitHub (apache/beam)
Resources & Compatibility
Minimum requirements: 8 GB RAM. ARM (e.g. Raspberry Pi) is not supported.
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Install and sign in to Beam CLI Install the Beam CLI with uv tool install beam-client, then run beam configure default --token YOUR_TOKEN using your Beam API token (a secret key). Without the saved token, Beam commands cannot create or control containers, so your job server will not start. source
- Self-host: run the local setup wizard If you are self-hosting on your own machine, run ./beta9 and answer the prompts for context name, gateway host, gateway port (often 1993), and token. If the port or token is wrong, your machine cannot receive work, so Beam may appear unresponsive or time out. source
- Expose the port your app uses If you define the job server as a Beam Pod, add its listening port to the ports list, such as ports=[8000] for an HTTP endpoint. Without the port exposed, the container may be running but you will not be able to reach the server from your browser or other tools. source
Fetched 53 days ago - each note links to its source
Variants
Alternatives (7)
- BunkerWeb - BunkerWeb is a self-hosted, open-source web application firewall and NGINX-based web server that protects services with secure-by-default s…
- Caddy - Caddy is an extensible web server and reverse proxy that serves Transport Layer Security (TLS) by default, making it simple to secure and r…
- Corvus - Self-hosted reverse proxy and web application firewall with a web console; it provides domain forwarding, free SSL, live traffic monitoring…
- HFS 2 (HTTP File Server) - You can use HFS (HTTP File Server) to send and receive files.
- Microsoft IIS - Internet Information Services (IIS) for Windows® Server is a flexible, secure and easy-to-manage Web server for hosting anything on the Web.
- lighttpd - Lighttpd is a secure, lightweight web server that uses memory and CPU more efficiently than comparable servers, with FastCGI, CGI, authenti…
- nginx - Nginx is an HTTP web server, reverse proxy, content cache, load balancer, TCP/UDP proxy, and mail proxy server.
Checked 40 days ago - source: AlternativeTo
Common questions
Why might a Python Beam pipeline fail when submitted to Spark from the job server?
It can be caused by a mismatch between the Spark client version in the job server and the Spark version of the cluster you submit to.
Which Spark master and job-server ports are referenced for local Beam portable runner testing with Spark?
The job-server ports 8097, 8098, and 8099 and the Spark master port 7077 are referenced for local testing.
Why won’t my Beam job server start after setting up the CLI?
Make sure you saved a valid Beam API token with the CLI. Without the saved token, Beam commands cannot create or control containers, and the job server will not start.
How are metrics reported when Beam runs on Spark?
The Spark runner uses Spark's built-in metrics system for Beam metrics reporting.
Answers sourced from beam.apache.org, docs.beam.cloud, stackoverflow.com, stackoverflow.com
Community
- Adding Spark Jobs in Standalone Mode to a Spring Boot REST server - Reddit (reddit)
- Running a python Apache Beam Pipeline on Spark - Stack Overflow (forum)
- It's possible to configure the Beam portable runner with the spark ... (forum)
Collected 48 days ago - public community threads