AWS Glue DPU configurations

1
votes

I see that the DPU is made of 4 vCPUs & 16 GB memory. Is it possible to change this settings for vCPU, memory, so that I don't run out of DPUs or exceed the DPU limit.

I think there is a maximum limit of 5 DPUs for a Dev Endpoint, and a maximum of 2 DEV Endpoints for an account?

Regards

Yuva

2
aws-glue

2 Answers

2
votes

Right now there is no way to configure DPU memory, but you can request a limit increase on your account to be able to use more DPUs.

1
votes

As of April 2019, there are two new types of workers:

You can now specify a worker type for Apache Spark jobs in AWS Glue for memory intensive workloads.

Previously, all Apache Spark jobs in AWS Glue ran with a standard configuration of 1 Data Processing Unit (DPU) per worker node and 2 Apache Spark executors per node. You can now pick from two new configurations, G.1X and G.2X, that provide more memory per executor.

https://aws.amazon.com/about-aws/whats-new/2019/04/aws-glue-now-supports-additional-configuration-options-for-memory-intensive-jobs/