# aws-glue-libs

aws-glue-libs is the official AWS Python library and Docker image for developing and testing AWS Glue PySpark ETL jobs locally. It is open source and self-hosted, installable on Docker Hub.

Canonical page: https://ourlocalhost.com/app/6695/aws-glue-libs

## Description

AWS Glue Libs is the official AWS Python library and Docker image for developing and testing AWS Glue PySpark ETL jobs locally. It mirrors the AWS Glue runtime so data engineers can write, debug, and run Spark code on their own machine before deploying to the cloud.

## Platform availability

Available on 0 of 10 platforms.

## Health score

60/100 (Good), based on 4 of 5 factors.
- Maintained: 68 (last commit 94 days ago)
- Popular: 57 (701 GitHub stars)
- Easy to install: 25 (0 install platforms, install notes, Docker image)
- Security check: 85 (no serious findings, CI)

## Links

- GitHub: https://github.com/awslabs/aws-glue-libs
- Docker Hub: https://hub.docker.com/r/amazon/aws-glue-libs

## Common questions

### Does aws-glue-libs include support for AWS Glue streaming jobs?

No. Glue streaming is supported in the separate aws-glue-streaming-libs repository.

### Do I need anything besides the awsglue Python package to run Glue code locally?

Yes. The awsglue package is only the Python interface to the Glue Spark runtime, so you also need the Glue ETL jar. It can be downloaded from the S3-backed Maven repository using Maven copy-dependencies.

### How can I run AWS Glue ETL code interactively on my own machine?

You can pull the amazon/aws-glue-libs Docker image to replicate the Glue runtime locally. You can also use the repository's bin executables to open a PySpark shell and run Glue Spark code interactively.

### Which Apache Spark distribution should I use for local development?

It depends on the Glue version: use Spark 2.4.3 for Glue 2.0, Spark 3.1.1 for Glue 3.0, Spark 3.3.0 for Glue 4.0, Spark 3.5.4 for Glue 5.0, Spark 3.5.6 for Glue 5.1, and Spark 4.1.1 for Glue 6.0.

### Can I unit test the PySpark parts of AWS Glue code before deploying?

Yes. PySpark parts can be unit tested locally, and dependencies can be zipped and uploaded to S3 for Glue deployment.

---
Source: Our Local Host (https://ourlocalhost.com/app/6695/aws-glue-libs), the self-hosted and open-source app directory.
