We have an AWS Glue job that is pulling from the a dynamodb table which is set to on-demand capacity. However, once we changed the table to on-demand, the glue job is taking forever to complete.
Presumably the glue job is trying to use a portion of the available read capacity...but this doesn't make sense with the new capacity model.
We are hoping to move all of our tables to the new on-demand capacity setting, but this would be a blocker for us.
Any ideas?
"When the DynamoDB table is in on-demand mode, AWS Glue handles the read capacity of the table as 40000. For exporting a large table, we recommend switching your DynamoDB table to on-demand mode."
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html
The below is no longer true
As per AWS Documentation On-demand is currently not supported by AWS Data Pipeline, the DynamoDB import/export tool, and AWS Glue. So you need to carefully chose which tables you want to move to new on-demand capacity.
