How do I copy data from AWS S3 to EC2 in a CloudFormation template?

Question

I've created a CloudFormation template that launches an AutoScaling group. During the launch, a policy allowing s3:GetObject access is attached to each EC2 instance. After this, I use User Data to install an Apache web server and PHP, and then change the settings for the relevant folders. I then need to copy multiple files from an S3 bucket (which has no public access) to the /var/www/html folder in each instance, but I can't work out how to do so without reverting to manually copying or syncing the files with the CLI after the CloudFormation stack has completed - this has to be an entirely automated process.

The user data in the template is as follows:

"UserData": {
    "Fn::Base64": {
        "Fn::Join": [
            "\n",
            [
                "#!/bin/bash",
                "yum update -y",
                "yum install -y httpd24 php56",
                "service httpd start",
                "chkconfig httpd on",
                "groupadd DMO",
                "usermod -a -G DMO ec2-user",
                "chgrp -R DMO /var/www",
                "chmod 2775 /var/www",
                "find /var/www -type d -exec chmod 2775 {} +",
                "find /var/www -type f -exec chmod 0664 {} +"
            ]
        ]
    }
}

Consider docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… — kichik

parsifal parsifal · Accepted Answer · 2020-03-05T19:44:52

To stick with that you're already doing, you could run the AWS CLI from within your userdata script:

"UserData": {
    "Fn::Base64": {
        "Fn::Join": [
            "\n",
            [
                "#!/bin/bash",
                "yum update -y",
                "yum install -y httpd24 php56",
                "service httpd start",
                "chkconfig httpd on",
                "groupadd DMO",
                "usermod -a -G DMO ec2-user",
                "chgrp -R DMO /var/www",
                "chmod 2775 /var/www",
                "aws s3 cp s3://MYBUCKET/MYFILE.zip /tmp",
                "unzip -d /var/www /tmp/MYFILE.zip",
                "rm /tmp/MYFILE.zip",
                "find /var/www -type d -exec chmod 2775 {} +",
                "find /var/www -type f -exec chmod 0664 {} +"
            ]
        ]
    }
}

In order to do this, you EC2 instance profile must grant permission to read the file from S3.

An alternative is to use AWS::CloudFormation::Init: it's a predefined metadata key that you can attach to either an EC2::Instance or AutoScaling::LaunchConfiguration resource, which allows you to configure packages, services, and individual files (including retrieving and unzipping a file from S3).

There's a tutorial here

How do I copy data from AWS S3 to EC2 in a CloudFormation template?

1 Answers