Resolving cyclical dependencies between AWS CDK CloudFormation stacks

Question

Context, I have a CDK app with two stacks using the following setup:

Stack_A:
    StateMachine_A
    Lambda_A
    S3Bucket_A
    IAMRole_A

Stack_B:
    StateMachine_B
    SageMakerTrainJob_B
    IAMRole_B

StateMachine_A runs Lambda_A using execution role IAMRole_A. A separate step in StateMachine_A writes data to S3Bucket_A. StateMachine_B runs SageMakerTrainJob_B using execution role IAMRole_B. Lambda_A's purpose is to start execution of StateMachine_B, whose SageMakerTrainJob_B needs to read from S3Bucket_A. Therefore, we have to configure the following permissions:

IAMRole_A needs startExecution permissions on StateMachine_B.
IAMRole_B needs read permissions on S3Bucket_A.

We tried to model this in CDK by creating a direct dependency in Stack_B on Stack_A, using references to IAMRole_A and S3Bucket_A within Stack_B's definition to grant the needed permissions in code. However, this generated the following error:

Error: 'Stack_B' depends on 'Stack_A' (dependency added using stack.addDependency()). Adding this dependency (Stack_A -> Stack_B/IAMRole_B/Resource.Arn) would create a cyclic reference.

Likewise, trying to model the dependency in the other direction gave the same error:

Error: 'Stack_A' depends on 'Stack_B' (dependency added using stack.addDependency()). Adding this dependency (Stack_B -> Stack_A/S3Bucket_A/Resource.Arn) would create a cyclic reference.

Is there any way around this using code dependencies? Are there any recommended best practices for situations like this? Some options we've considered include:

Using a third stack that depends on both to give Stack_A and Stack_B access to each other's resources.
Creating additional access roles for the necessary resources within each stack and maintaining assumeRole permissions for the Lambda/SageMaker roles somewhere outside of CDK.
Putting them all in one stack. Not great for organization and makes the resources really tightly coupled--we may not want StateMachine_A to be the only entry point to StateMachine_B in the future.

Also , I see there were similar-sounding issues during CDK development with CodeCommit/CodePipeline and APIGateway/Lambda. Is this a related bug, or are we just trying to do something that's not supported?

Based on your description, it seems like S3Bucket_A is not directly depended on by any other A constructs, nor does S3Bucket_A have a direct dependency on any of the other A constructs. Is that correct? — Matthew Pope
I left out a step in StateMachine_A that writes to S3Bucket_A for simplicity. I edited the description to reflect that. — user2446256

Jason Wadsworth Jason Wadsworth · Accepted Answer · 2020-02-26T16:57:39

Circular references are always tricky. This isn't a problem that is unique to the CDK. As you explain the problem logically you see where things start to break down. CloudFormation must create any resources that another resource depends on before it can create the dependent resource. There isn't a one solution fits all approach to this, but I'll give some ideas that work.

Promote shared resources to another stack. In your case the S3 bucket needs to be used by both stacks, so if that is in a stack that runs before both you can create the S3 bucket, use an export/import in stack B to reference the S3 bucket, and use and export/import in stack A to reference both the S3 bucket and the state machine in B.
Use wildcards in permissions. Often you can know the name, or enough of the name, of a resource to use wildcards in your permissions. You want to keep the permissions tightly scoped, but quite often a partial name match is good enough. Use this option with caution, of course. Also keep in mind that many resources can be named by you. Many prefer not to do this ever, and some resources you shouldn't (like and S3 bucket), but I often find it's easier to name things.
Create a custom resource to tie things together. If you have a true circular dependency that cannot be resolved (even in the same stack) you may need to use a custom resource to do the work for you. A prime example of this is S3 bucket events.

Resolving cyclical dependencies between AWS CDK CloudFormation stacks

1 Answers