AWS CloudFormation: Nested Sub with Dynamic References using {{resolve}} causes error and doesn't execute resolve to get value from Parameter Store

Question

I am trying to use AWS CloudFormation Template to create an EC2 Instance with some userdata generated using dynamic references and cross-stack reference in the template . There is a parameter stored in AWS Systems Manager Parameter Store with Name:/MyCustomParameter and Value:Test1.

The idea is to pass a parameter to the template stack (Stack A) which refers to another cloudformation stack (StackB). Stack B exports a variable with reference "StackB::ParameterStoreName". Stack A uses Fn::ImportValue: 'StackB::ParameterStoreName' to get it's value so that it can be used with dynamic references method to get it's value from AWS SSM Parameter Store using {{resolve:ssm:/MyCustomParameter:1}} and pass it's value to the UserData field in the template. I am facing difficulties while trying to use nested Fn::Sub: function with this use-case.

I tried removing the | pipe and using double quotes with escaped new line character but that doesn't work.

I also tried using a different type of resource and it's properties where is worked. Below is an example of the code that worked.

Resources:
  TestBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: 
        Fn::Sub:
          - '${SSMParameterValue}-12345'
          - SSMParameterValue: 
              Fn::Sub:
                - '{{resolve:ssm:${SSMParameterName}:1}}'
                - SSMParameterName: 
                    Fn::ImportValue:
                      !Sub '${CustomStack}::ParameterStoreName'

Below is an extract of the current code I have:

Parameters:
  CustomStack:
    Type: "String"
    Default: "StackB"
Resources:
  MyCustomInstance:  
    Type: 'AWS::EC2::Instance'
        Properties:
        UserData:
            Fn::Base64:
            Fn::Sub:
                - |
                #!/bin/bash -e 
                #
                # Bootstrap and join the cluster 
                /etc/eks/bootstrap.sh --b64-cluster-ca '${SSMParameterValue}' --apiserver-endpoint '${Endpoint}' '${ClusterName}'"
                - SSMParameterValue:
                    Fn::Sub:
                    - '{{resolve:ssm:/${SSMParameterName}:1}}'
                    - SSMParameterName: 
                        Fn::ImportValue:
                            !Sub '${CustomStack}::ParameterStoreName'
                Endpoint:
                    Fn::ImportValue:
                    !Sub '${CustomStack}::Endpoint'
                ClusterName: 
                    Fn::ImportValue:
                    !Sub '${CustomStack}::ClusterStackName'

Current Output:

#!/bin/bash -e 
# 
# Bootstrap and join the cluster 
/etc/eks/bootstrap.sh --b64-cluster-ca `{{resolve:ssm:MyCustomParameter:1}}` --apiserver-endpoint 'https://04F1597P0HJ11FQ54K0YFM9P19.gr7.us-east-1.eks.amazonaws.com' 'eks-cluster-1'

Expected Output:

#!/bin/bash -e 
# 
# Bootstrap and join the cluster 
/etc/eks/bootstrap.sh --b64-cluster-ca `Test1` --apiserver-endpoint 'https://04F1597P0HJ11FQ54K0YFM9P19.gr7.us-east-1.eks.amazonaws.com' 'eks-cluster-1'

Max Allan Max Allan · Accepted Answer · 2019-12-10T22:18:26

I think it is because the resolve is in the base64, maybe...? When it processes the line it just sees a block of base64 and not the {{resolve...}} code. The "resolves" get processed at a later pass than the !Functions, because they can't be resolved until the code is running.

To work around it, I added a temporary SSM parameter :

eksCAtmp:
  Type: "AWS::SSM::Parameter"
  Properties:
    Type: String
    Value:
      Fn::Join:
        - ''
        - - '{{resolve:ssm:'
          - Fn::ImportValue:
              !Sub "${ClusterName}-EksCA"
          - ':1}}'

That imports the original SSM parameter and gets rid of the requirement to "import" and resolve it again. So now you can use !GetAtt eksCAtemp.Value

eg:

  UserData: !Base64
    "Fn::Sub":
      - |
        #!/bin/bash
        set -o xtrace
        /etc/eks/bootstrap.sh ${ClusterName} --b64-cluster-ca ${CA}  --apiserver-endpoint ${endpoint} --kubelet-extra-args '--read-only-port=10255'
        /opt/aws/bin/cfn-signal --exit-code $? \
                 --stack  ${AWS::StackName} \
                 --resource NodeGroup  \
                 --region ${AWS::Region}
      - endpoint:
          Fn::ImportValue:
            !Sub "${ClusterName}-EksEndpoint"
        CA: !GetAtt eksCAtmp.Value

(Of course if they allowed cross stack exports to be more than 1024 characters, we wouldn't need this for firing up EKS on a private network.)

AWS CloudFormation: Nested Sub with Dynamic References using {{resolve}} causes error and doesn't execute resolve to get value from Parameter Store

2 Answers