Iterating over results from previous state in AWS Step Functions

Question

I need to develop a 'State Machine' using 'AWS Step Functions' that accomplishes the following:

Call a Lambda function that will connect to DynamoDb & retrieve a list of rows. (I know how to do this.)
For each row from previous step, I need to call another Lambda function until all rows are read.

How do I do step #2 above in AWS Step Functions? In other words, how do I iterate over the results from the previous step.

Take a look at this Step Functions Iterator pattern. You may be able to learn something from it. — Matt Houser
Yes, quick Googling directed me to this link but I was wondering if this is the only way to do it. It seems like a workaround. Is that because this functionality doesn't exist out of the box? — DilTeam
Have a look at github.com/eclecticlogic/stepper This translates a high level language (Stepper - java/javascript like) into step-function state machine with support for iteration. — Καrτhικ

tgoodhart tgoodhart · Accepted Answer · 2018-05-11T15:58:17

It's not pretty, but this can be accomplished out of the box / without an iterator lambda by using JSONPath's slice operator and a Sentinel value.

Here's an example state machine:

{
  "Comment": "Example of how to iterate over an arrray of items in Step Functions",
  "StartAt": "PrepareSentinel",
  "States": {
    "PrepareSentinel": {
      "Comment": "First, prepare a temporary array-of-arrays, where the last value has a special SENTINEL value.",
      "Type": "Pass",
      "Result": [
        [
        ],
        [
          "SENTINEL"
        ]
      ],
      "ResultPath": "$.workList",
      "Next": "GetRealWork"
    },
    "GetRealWork": {
      "Comment": "Next, we'll populate the first array in the temporary array-of-arrays with our actual work. Change this from a Pass state to a Task/Activity that returns your real work.",
      "Type": "Pass",
      "Result": [
        "this",
        "stage",
        "should",
        "return",
        "your",
        "actual",
        "work",
        "array"
      ],
      "ResultPath": "$.workList[0]",
      "Next": "FlattenArrayOfArrays"
    },
    "FlattenArrayOfArrays": {
      "Comment": "Now, flatten the temporary array-of-arrays into our real work list. The SENTINEL value will be at the end.",
      "Type": "Pass",
      "InputPath": "$.workList[*][*]",
      "ResultPath": "$.workList",
      "Next": "GetNextWorkItem"
    },
    "GetNextWorkItem": {
      "Comment": "Extract the first work item from the workList into currentWorkItem.",
      "Type": "Pass",
      "InputPath": "$.workList[0]",
      "ResultPath": "$.currentWorkItem",
      "Next": "HasSentinelBeenReached"
    },
    "HasSentinelBeenReached": {
      "Comment": "Check if the currentWorkItem is the SENTINEL. If so, we're done. Otherwise, do something.",
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.currentWorkItem",
          "StringEquals": "SENTINEL",
          "Next": "Done"
        }
      ],
      "Default": "DoWork"
    },
    "DoWork": {
      "Comment": "Do real work using the currentWorkItem. Change this to be an activity/task.",
      "Type": "Pass",
      "Next": "RemoveFirstWorkItem"
    },
    "RemoveFirstWorkItem": {
      "Comment": "Use the slice operator to remove the first item from the list.",
      "Type": "Pass",
      "InputPath": "$.workList[1:]",
      "ResultPath": "$.workList",
      "Next": "GetNextWorkItem"
    },
    "Done": {
      "Type": "Succeed"
    }
  }
}

Iterating over results from previous state in AWS Step Functions

1 Answers