AWS CloudFormation 'UserData' Doesn't seem to work

Question

I'm writing an AWS CloudFormation script to build an EC2 instance. I'd like to provision the instance by installing some packages, downloading some repos and running some scripts. Amazon tells me I can do this in CloudFormation with the UserData field. However, it just doesn't seem to work at all.

Here is what i'm working with currently:

DWHServer:
Type: "AWS::EC2::Instance"
Properties:
  DisableApiTermination: false # no termination protection
  EbsOptimized: false # optimize for elastic block store
  IamInstanceProfile: !Ref DWHServerIAMIP
  ImageId: "ami-5189a661" # ubunty-trusty-14.04-amd64-server-20150325
  InstanceInitiatedShutdownBehavior: "terminate"
  InstanceType: "t2.medium"
  KeyName: !FindInMap [EnvMap, KeyPair, !Ref EnvType]
  Monitoring: true
  SecurityGroupIds:
    - !Ref DWHServerSG
  SourceDestCheck: true # ??
  SubnetId: "subnet-aed2ecf6" # Stage-etl-2c
  UserData: !Base64
    "Fn::Join": ["", ["#!/bin/bash -xe\n", "touch ~/confirm_work.txt\n"]]

This is the most simple example. I just want it to make a file to prove that it's running. But it doesn't even do that. The docs say to look at something called /var/log/cloud-init-output.log. I looked there, but don't see anything about UserData. There does seem to be some sort of network error, but I'm not sure how to interpret it or what to do about it.

Here are the contents of the cloud-init-output.log file on the instance:

Cloud-init v. 0.7.5 running 'init-local' at Sat, 04 Mar 2017 02:40:07 +0000. Up 3.85 seconds.
Cloud-init v. 0.7.5 running 'init' at Sat, 04 Mar 2017 02:40:09 +0000. Up 6.01 seconds.
ci-info: +++++++++++++++++++++++++Net device info+++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | Device |  Up  |  Address  |      Mask     |     Hw-Address    |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: |   lo   | True | 127.0.0.1 |   255.0.0.0   |         .         |
ci-info: |  eth0  | True | 10.0.7.84 | 255.255.255.0 | 0a:3a:b0:a4:96:5d |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway  |    Genmask    | Interface | Flags |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 10.0.7.1 |    0.0.0.0    |    eth0   |   UG  |
ci-info: |   1   |   10.0.7.0  | 0.0.0.0  | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
Mar  4 02:40:11 ubuntu pollinate[723]: ERROR: Network communication failed [60]\n02:40:10.394529 * Hostname was NOT found in DNS cache
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     002:40:10.407240 *   Trying 91.189.94.24...
02:40:10.550022 * Connected to entropy.ubuntu.com (91.189.94.24) port 443 (#0)
02:40:10.551661 * successfully set certificate verify locations:
02:40:10.551698 *   CAfile: /etc/pollinate/entropy.ubuntu.com.pem
CApath: /dev/null
02:40:10.551804 * SSLv3, TLS handshake, Client hello (1):
02:40:10.551832 } [data not shown]
02:40:10.711080 * SSLv3, TLS handshake, Server hello (2):
02:40:10.711129 { [data not shown]
02:40:10.711191 * SSLv3, TLS handshake, CERT (11):
02:40:10.711216 { [data not shown]
02:40:10.711490 * SSLv3, TLS alert, Server hello (2):
02:40:10.711520 } [data not shown]
02:40:10.711602 * SSL certificate problem: unable to get local issuer certificate
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
02:40:10.711732 * Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
2017-03-04 02:40:11,144 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random.pyc'>) failed
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
0c:54:09:ab:bc:b8:63:b5:6c:d2:d5:47:21:4a:38:6f root@ip-10-0-7-84
The key's randomart image is:
+--[ RSA 2048]----+
|     .oo..       |
|    o...o .      |
|     +o. . .     |
|   . .Eo  .      |
|    o. .S.       |
|   .... . .      |
|  .+.o   .       |
|  +.=            |
| ..+             |
+-----------------+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
89:26:94:17:79:6d:45:15:fc:5f:37:95:31:2e:e9:f7 root@ip-10-0-7-84
The key's randomart image is:
+--[ DSA 1024]----+
|      .. . oooo+o|
|     .... o   +.o|
|    o .. .   o o.|
|   . . . .  . ..+|
|    . o S    . .=|
|     o        . o|
|                E|
|                 |
|                 |
+-----------------+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
The key fingerprint is:
af:a2:c7:b3:95:5c:17:2e:ce:69:b3:f6:39:c7:67:91 root@ip-10-0-7-84
The key's randomart image is:
+--[ECDSA  256]---+
|                 |
|                 |
|            .    |
|           . .   |
|        S o o   .|
|       . * +   E |
|     .  + B  .  .|
|      =. o.o..o o|
|    .o.+....oo o |
+-----------------+
Cloud-init v. 0.7.5 running 'modules:config' at Sat, 04 Mar 2017      02:40:14 +0000. Up 11.53 seconds.
Generating locales... en_US.UTF-8... up-to-date
Generation complete.
Cloud-init v. 0.7.5 running 'modules:final' at Sat, 04 Mar 2017          02:40:17 +0000. Up 13.61 seconds.
+ touch /root/confirm_work.txt
Cloud-init v. 0.7.5 finished at Sat, 04 Mar 2017 02:40:17 +0000.     Datasource DataSourceEc2.  Up 13.83 seconds

Any tips would be greatly appreciated. Thanks!

Did you have a look at the instance's system log? From the EC2 console: select your instance, then Actions -> Instance Settings -> Get System Log — spg
I looked at the system log, and the cloud-init-output.log file presents the same information after bootup. Thanks for the suggestion though! — bgenchel

Pierre Pierre · Accepted Answer · 2021-10-14T23:57:36

Look at the second to last entry in the log:

+ touch /root/confirm_work.txt

The command is indeed invoked. Note that all commands in your EC2 user data will show up in that log file (/var/log/cloud-init-output.log) with a PLUS sign prepended to it (like above). Is it possible that the touch command is not there? That would be surprising. But add an "echo" command before the touch, you should see the output and that would confirm that it's all working fine. Maybe you're trying to touch in a directory you don't have access. Maybe try to touch a file in /tmp to narrow things down...

AWS CloudFormation 'UserData' Doesn't seem to work

2 Answers