aws-glue-libs

AWS Glue Libs is the official AWS Python library and Docker image for developing and testing AWS Glue PySpark ETL jobs locally.

  • Databases
  • Development
  • Tools & Utilities
AWS Glue Libs is the official AWS Python library and Docker image for developing and testing AWS Glue PySpark ETL jobs locally. It mirrors the AWS Glue runtime so data engineers can write, debug, and run Spark code on their own machine before deploying to the cloud.

Platform availability

Available on 0 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

60/100 Good

Maintained
68last commit 94 days ago
Popular
57701 GitHub stars
Easy to install
250 install platforms, install notes, Docker image
Light to run
No data
Security check
85no serious findings, CI

Based on 4 of 5 factors.

GitHub stars
701
Open issues
147
Last commit
2026-08-19
Latest release
v4.0
Activity
stale

Checked yesterday - source: GitHub (awslabs/aws-glue-libs)

Resources & Compatibility

ARM (e.g. Raspberry Pi) is supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Running locally Pull the amazon/aws-glue-libs image from Docker Hub and run it to replicate the AWS Glue runtime on your own machine, or use the repo's bin/ executables to open a PySpark shell and run your ETL code interactively. source

Fetched 49 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

Does aws-glue-libs include support for AWS Glue streaming jobs?

No. Glue streaming is supported in the separate aws-glue-streaming-libs repository.

Do I need anything besides the awsglue Python package to run Glue code locally?

Yes. The awsglue package is only the Python interface to the Glue Spark runtime, so you also need the Glue ETL jar. It can be downloaded from the S3-backed Maven repository using Maven copy-dependencies.

How can I run AWS Glue ETL code interactively on my own machine?

You can pull the amazon/aws-glue-libs Docker image to replicate the Glue runtime locally. You can also use the repository's bin executables to open a PySpark shell and run Glue Spark code interactively.

Which Apache Spark distribution should I use for local development?

It depends on the Glue version: use Spark 2.4.3 for Glue 2.0, Spark 3.1.1 for Glue 3.0, Spark 3.3.0 for Glue 4.0, Spark 3.5.4 for Glue 5.0, Spark 3.5.6 for Glue 5.1, and Spark 4.1.1 for Glue 6.0.

Can I unit test the PySpark parts of AWS Glue code before deploying?

Yes. PySpark parts can be unit tested locally, and dependencies can be zipped and uploaded to S3 for Glue deployment.

Answers sourced from docs.aws.amazon.com, github.com, stackoverflow.com

Community

Collected 49 days ago - public community threads