How to see progress when using Glue to export DynamoDB table

Question

I'm trying to export every item in a DynamoDB table to S3. I found this tutorial https://aws.amazon.com/blogs/big-data/how-to-export-an-amazon-dynamodb-table-to-amazon-s3-using-aws-step-functions-and-aws-glue/ and followed the example. Basically,

table = glueContext.create_dynamic_frame.from_options(
  "dynamodb",
  connection_options={
    "dynamodb.input.tableName": table_name,
    "dynamodb.throughput.read.percent": read_percentage,
    "dynamodb.splits": splits
  }
)

glueContext.write_dynamic_frame.from_options(
  frame=table,
  connection_type="s3",
  connection_options={
    "path": output_path
  },
  format=output_format,
  transformation_ctx="datasink"
)

I tested it in a tiny table in nonprod environment and it works fine. But my Dynamo table in production is over 400GB, 200 mil items. I suppose it'll take a while, but I have no idea how long to expect. Hours, or even days? Are there any way to show progress? For example, showing a count of how many items have been processed. I don't want to blindly start this job and wait.

Answer 1

One way would be to enable continuous logging for your AWS Glue Job to monitor its progress.

Another way would be to trigger a Lambda function whenever a file has been stored in S3, using Amazon S3 event notifications.

Answer 2

Did you try the custom waiter class within was docs? For instance custom waiter for a Glue Job should look something like this:

class JobCompleteWaiter(CustomWaiter):
    def __init__(self, client):
        super().__init__(
            "JobComplete",
            "get_job_run",
            "JobRun.JobRunState",
            {"SUCCEEDED": WaitState.SUCCEEDED, "FAILED": WaitState.FAILED},
            client,
            max_tries=100,
        )

    def wait(self, JobName, RunId):
        self._wait(JobName=JobName, RunId=RunId)

According to boto3 docs, you should expect a set of 6 different possible states from a JOB: STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT' So I chost checkein whether was SUCCEEDED or FAILED.