I am not able to copy S3 file to EC2 instance using Userdata in CloudFormation

Question

I am unable to download an S3 file to my EC2 instance using CloudFormation Userdata property. I have assigned an IAM role but still not able to get it resolved.

I assigned the role inside the template.

I tried passing Access Key and Secret Access Key - same result.

"Parameters": {       
    "VpcId": {
        "Type": "AWS::EC2::VPC::Id",
        "Description": "Id of an existing VPC to use for "
    },
    "SubnetId": {
        "Type": "AWS::EC2::Subnet::Id",
        "Description": "Id of an existing subnet id to use for "
    },

    "SecurityGroupIds": {
        "Description": "Security groups ",
        "Type": "List<AWS::EC2::SecurityGroup::Id>",
        "ConstraintDescription": "using existing security  be list of EC2 security group ids"
    },
    "instanceType": {
        "Type": "String",
        "Default": "t2.micro",
        "AllowedValues": [
            "t2.micro"

        ],
        "Description": "Enter Instance Type "
    },
    "AWSREGION": {
        "Type": "String",
        "Default": "us-east-1",
        "AllowedValues": [
            "us-east-1"
        ],
        "Description": "Enter AWS_REGION."
    }
},
"Resources": {
    "InstanceRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": { "Service": [ "ec2.amazonaws.com" ] },
              "Action": [ "sts:AssumeRole" ]
            }
          ]
        },
        "Path": "/",
        "Policies": [
          { 
            "PolicyName": "S3_Access",
            "PolicyDocument": {
              "Statement": [
                {
                  "Effect": "Allow",
                 "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:DeleteObject"
                  ],
                  "Resource": ["arn:aws:s3:::mybucketlocation/*"]
                }
              ]
            }
          }
        ]
      }
    },
    "InstanceProfile": {
      "Type": "AWS::IAM::InstanceProfile",
      "Properties": {
        "Path": "/",
        "Roles": [ { "Ref": "InstanceRole" }
        ]
      }
    },
    "EdgeNode": {
        "Type": "AWS::EC2::Instance",
        "Properties": {
            "IamInstanceProfile": { "Ref": "InstanceProfile" },
            "InstanceType": { "Ref" : "instanceType" },
            "ImageId": "ami-0cc96feef8c6bbff3",
            "SubnetId": { "Ref" : "SubnetId" },
            "KeyName": "my-key",
            "SecurityGroupIds": {
                "Ref": "SecurityGroupIds"
            },
            "UserData": {
                "Fn::Base64": {
                    "Fn::Join": [
                        "\n",
                        [
                            "#!/bin/bash",
                            "echo \"\" > /home/xyz/index.txt", 
                            {
                                "Fn::Join": [
                                    "",
                                    [
                                        "echo \"AWS_REGION: ",
                                        {
                                            "Ref": "AWSREGION"
                                        },
                                        "\" >> /home/xyz/index.txt"
                                    ]
                                ]

                            },
                            {
                                 "Fn::Join": ["", [
                                    "<script>\n",
                                    "cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
                                    "</script>"
                                  ] ] 
                            }



                        ]
                    ]

                }
            }
        },

        "Metadata": {
            "AWS::CloudFormation::Init": {
              "config": {
                "commands" : {
                  "Pullcode" : {
                    "command" : "aws s3 sync s3://mybucketlocation /home/xyz/ --debug"
                  }
                }
              } 
            },
            "AWS::CloudFormation::Designer": {
                "id": "e37a9183-9f81c2fbd39"


            }
        }
    }
}

In cloud-init-output.log I got this:

/var/lib/cloud/instance/scripts/part-001: line 7: syntax error near unexpected token newline' /var/lib/cloud/instance/scripts/part-001: line 7:' Jun 21 11:45:05 cloud-init[4071]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [2] Jun 21 11:45:05 cloud-init[4071]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts) Jun 21 11:45:05 cloud-init[4071]: util.py[WARNING]: Running module scripts-

John Rotenstein John Rotenstein · Accepted Answer · 2019-06-22T01:40:59

These lines seem strange:

                             "Fn::Join": ["", [
                                "<script>\n",
                                "cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
                                "</script>"

You are launching an Amazon EC2 instance. However, these lines look like they were taken from User Data for a Windows instance.

Also, you are prompting the user for a Region, but the script is already running in a specific region, so you can use { "Ref" : "AWS::Region" } to access the value.

You probably want your User Data script to look like this:

"UserData": {
    "Fn::Base64": {
        "Fn::Join": [
            "\n",
            [
                "#!/bin/bash",
                {
                    "Fn::Sub": "echo AWS_REGION: ${AWS::REGION} >>/home/xyz/index.txt"
                },
                {
                    "Fn::Sub": "cfn-init -v -s ${AWS::StackId} -r EdgeNode --region ${AWS::Region}"
                },
            ]
        ]

    }
}

I didn't test it, so you might need to tweak some things.

I am not able to copy S3 file to EC2 instance using Userdata in CloudFormation

2 Answers