How should I deploy my aws state machines through cloudformation?

Question

I am using cloudformation to update my aws stack. I have several state machines. I update them with cloudformation as well and get their ARN's and put them as Environment variables to lambda function again in my cloud formation stack.

However question is this: When I change my step functions and update my stack, cloudformation is going to delete old state machines and create new ones with random names. I can get the ARN and use them in my lambda, that's not the problem but my old state machines are going get deleted. This means that any running executions are gonna get stopped by next state transition. I might have long running state machine executions. It looks like I cannot do this in a prod environment.

Any suggestions?

EDIT:

I am using AWS Step Functions for my state machines. This question is all about AWS Step Functions and Cloudformation. How to update step functions with running executions with cloudformation.

Answer 1

I guess I found out how this works.

Documentation for DeleteStateMachine says:

Deletes a state machine. This is an asynchronous operation-- it sets the state machine's status to "DELETING" and begins the delete process. Each state machine execution will be deleted the next time it makes a state transition. After all executions have completed or been deleted, the state machine itself will be deleted.

This made me think that when updating a state machine with cloudformation it was going to delete a running state machine right after the next state transition for each execution. I guess this is not the case. Because I tried replacing a long running state machine and now it says:

Deleting. The deletion operation will not complete while any Executions are In Progress. Consider stopping any long-running Executions via the console, API, or command line.

This kind of conflicts with what documentation says about deleting a state machine. However I guess Cloudformation does not use same Delete operation as referred in documentation.

It would be good to have a clarification from an AWS expert on this matter.

Answer 2

My recent lessons with CloudFormation + Step Functions is to always first create the state machine with boto3 or any other SDK because CloudFormation will not tell you if you are missing any permissions. boto3 calls will fail immediately with a pretty clear error message. Whereas CloudFormation (CFN) will just try and retry for almost an hour and then even fail to roll back properly. I think the CFN implementation of State Machine update is equally bad. My suggestion is to always delete a machine and then create a new one with a new name. This is quite typical with CFN resources that updates don't really work in practical scenarios because the deletions are asynchronous and I suspect the CFN implementation does not account for that and when a resource needs to be replaced it just attempts to create a new one straight away with the same name and obviously fails. There is no open CFN bug list so all this is speculation. I went to AWS Summit earlier this week, talked with an AWS Architect (AWS employee) and he was just blushing.

This is a IAM role for which my state machine creation did not fail:

StateMachineRole:
    Type: AWS::IAM::Role
    Properties:
        AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Principal:
                Service:
                - !Sub "states.${AWS::Region}.amazonaws.com"
            Action: sts:AssumeRole
        ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaRole
        - arn:aws:iam::aws:policy/AWSStepFunctionsFullAccess
        Policies:
        - PolicyName: StateMachineAccessPolicy
            PolicyDocument:
            Statement:
                - Effect: Allow
                Action:
                    - events:*
                Resource: "*"