aws-glue-libs
AWS Glue Libs is the official AWS Python library and Docker image for developing and testing AWS Glue PySpark ETL jobs locally.
Platform availability
Available on 0 of 10 platforms
- Unraid (not listed)
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
60/100 Good
- Maintained
- 68last commit 94 days ago
- Popular
- 57701 GitHub stars
- Easy to install
- 250 install platforms, install notes, Docker image
- Light to run
- No data
- Security check
- 85no serious findings, CI
Based on 4 of 5 factors.
- GitHub stars
- 701
- Open issues
- 147
- Last commit
- 2026-08-19
- Latest release
- v4.0
- Activity
- stale
Checked yesterday - source: GitHub (awslabs/aws-glue-libs)
Resources & Compatibility
ARM (e.g. Raspberry Pi) is supported.
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Running locally Pull the amazon/aws-glue-libs image from Docker Hub and run it to replicate the AWS Glue runtime on your own machine, or use the repo's bin/ executables to open a PySpark shell and run your ETL code interactively. source
Fetched 49 days ago - each note links to its source
Variants
Alternatives
No alternatives collected yet.
Common questions
Does aws-glue-libs include support for AWS Glue streaming jobs?
No. Glue streaming is supported in the separate aws-glue-streaming-libs repository.
Do I need anything besides the awsglue Python package to run Glue code locally?
Yes. The awsglue package is only the Python interface to the Glue Spark runtime, so you also need the Glue ETL jar. It can be downloaded from the S3-backed Maven repository using Maven copy-dependencies.
How can I run AWS Glue ETL code interactively on my own machine?
You can pull the amazon/aws-glue-libs Docker image to replicate the Glue runtime locally. You can also use the repository's bin executables to open a PySpark shell and run Glue Spark code interactively.
Which Apache Spark distribution should I use for local development?
It depends on the Glue version: use Spark 2.4.3 for Glue 2.0, Spark 3.1.1 for Glue 3.0, Spark 3.3.0 for Glue 4.0, Spark 3.5.4 for Glue 5.0, Spark 3.5.6 for Glue 5.1, and Spark 4.1.1 for Glue 6.0.
Can I unit test the PySpark parts of AWS Glue code before deploying?
Yes. PySpark parts can be unit tested locally, and dependencies can be zipped and uploaded to S3 for Glue deployment.
Answers sourced from docs.aws.amazon.com, github.com, stackoverflow.com
Community
- run glue container locally using docker-compose (forum)
- AWS Glue - Reddit (reddit)
- How do you guys build with aws glue? : r/dataengineering - Reddit (reddit)
- python - Can I test AWS Glue code locally? - Stack Overflow (forum)
- Thoughts on AWS GLUE? : r/dataengineering - Reddit (reddit)
- Amazon/aws-glue-libs - Docker Hub - Docker Community Forums (forum)
Collected 49 days ago - public community threads