AWS Step Function get all Range keys with a single primary key in DynamoDB

Question

I'm building AWS Step Function state machines. My goal is to read all Items from a DynamoDB table with a specific value for the Hash key (username) and any (wildcard) Sort keys (order_id). Basically something that would be done from SQL with:

SELECT username,order_id FROM UserOrdersTable
WHERE username = 'daffyduck'

I'm using Step Functions to get configurations for AWS Glue jobs from a DynamoDB table and then spawn a Glue job via step function's Map for each dynamodb item (with parameters read from the database).

 "Read Items from DynamoDB": {
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:getItem",
  "Parameters": {
    "TableName": "UserOrdersTable",
    "Key": {
      "username": {
        "S": "daffyduck"
      },
      "order_id": {
        "S": "*"
      }
    }
  },
  "ResultPath": "$",
  "Next": "Invoke Glue jobs"
}

But I can't bring the state machine to read all order_id's for the user daffyduck in the step function task above. No output is displayed using the code above, apart from http stats. Is there a wildcard for order_id ? Is there another way of getting all order_ids? The query customization seems to be rather limited inside step functions: https://docs.amazonaws.cn/amazondynamodb/latest/APIReference/API_GetItem.html#API_GetItem_RequestSyntax

Basically I'm trying to accomplish what can be done from the command line like so:

$ aws dynamodb query \
--table-name UserOrdersTable \
--key-condition-expression "Username = :username" \
--expression-attribute-values '{
    ":username": { "S": "daffyduck" }
}'

Any ideas? Thanks

Do note that your CLI uses Query while your SFN Machine has GetItem(arn:aws:states:::dynamodb:getItem) in Resource section. Unfortunately according to SFN docs, DynamoDB's Operation Query is not supported. As (almost) always in AWS, if something isn't supported out of the box, you are able to fallback to Lambda function. What I mean is that you can just run a custom Lambda as a first Task to perform your DDB Call via DDB API and return the expected result to the state instead of trying to use built in SFN-DDB integration. — Никита Васильев

Balu Vyamajala Balu Vyamajala · Accepted Answer · 2021-01-18T20:57:57

I don't think that is possible with Step functions Dynamodb Service yet. currently supports get, put, delete & update Item, not query or scan. For GetItem we need to pass entire KEY (Partition + Range Key)

For the primary key, you must provide all of the attributes. For example, with a simple primary key, you only need to provide a value for the partition key. For a composite primary key, you must provide values for both the partition key and the sort key.

We need to write a Lambda function to query Dynamo and return a map and invoke the lambda function from step.

AWS Step Function get all Range keys with a single primary key in DynamoDB

1 Answers