Ansible: Trying to create multiple EC2 instances in multiple regions in one shot

Question

I'm trying to create an AWS EC2 ansible playbook that:

1) first allocates one VPC each on three Regions which are: us-west-1, ap-northeast-1 and eu-west-1.

2) Finds the latest ubuntu AMI for each region (ec2_ami_search),

3) then using the discovered results from 1) and 2), create one EC2 instance per region with the latest ubuntu AMI (for the region) with Availibility Zones us-west-1a, ap-northeast-1a and eu-west-1a, respectively.

With Ansible, I had no problem with step 1) and 2) which was simply:

> 

  tasks:
  - name: create a vpc
    ec2_vpc:
      state: present
      region: "{{ item.region }}"
      internet_gateway: True
      resource_tags: { env: production}
      cidr_block: 10.0.0.0/16
      subnets:
        - cidr: 10.0.0.0/24
          az: "{{ item.az }}"
          resource_tags:
            env: production
            tier: public
      route_tables:
        - subnets:
          - 10.0.0.0/24
          routes:
          - dest: 0.0.0.0/0
            gw: igw
    with_items:
      - region: us-west-1
        az: us-west-1a
      - region: ap-northeast-1
        az: ap-northeast-1a
      - region: eu-west-1
        az: eu-west-1a
...
  - name: Get the ubuntu trusty AMI
    ec2_ami_search: distro=ubuntu release=trusty virt=hvm region={{ item }}
    with_items:
      - us-west-1
      - ap-northeast-1
      - eu-west-1
    register: ubuntu_image
...
>

and the outputted for the ubuntu_image variable with debug module:

TASK: [print out ubuntu images] *********************************************** 
ok: [localhost] => {
    "ubuntu_image": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "aki": null, 
                "ami": "ami-b33dccf7", 
                "ari": null, 
                "changed": false, 
                "invocation": {
                    "module_args": "distro=ubuntu release=trusty virt=hvm region=us-west-1", 
                    "module_name": "ec2_ami_search"
                }, 
                "item": "us-west-1", 
                "serial": "20150629", 
                "tag": "release"
            }, 
            {
                "aki": null, 
                "ami": "ami-9e5cff9e", 
                "ari": null, 
                "changed": false, 
                "invocation": {
                    "module_args": "distro=ubuntu release=trusty virt=hvm region=ap-northeast-1", 
                    "module_name": "ec2_ami_search"
                }, 
                "item": "ap-northeast-1", 
                "serial": "20150629", 
                "tag": "release"
            }, 
            {
                "aki": null, 
                "ami": "ami-7c4b0a0b", 
                "ari": null, 
                "changed": false, 
                "invocation": {
                    "module_args": "distro=ubuntu release=trusty virt=hvm region=eu-west-1", 
                    "module_name": "ec2_ami_search"
                }, 
                "item": "eu-west-1", 
                "serial": "20150629", 
                "tag": "release"
            }
        ]
    }
}

However, I couldn't figure out how to make step 3) take the result from the ubuntu_image register variable and then determine which of the 3 AMIs and Subnets the given EC2 instance belonged. See below where as a workaround I manually hardcoded the ami and subnet value which I simply got from the printout from the above ubuntu_image printout:

  - name: start the instances
    ec2:
      image: "{{ item.ami }}"  # MANUALLY HARDCODED
      region: "{{ item.region }}"
      instance_type: "{{ instance_type }}"
      assign_public_ip: True
      key_name: "{{ item.name }}"
      group: ["http deployment", "ssh deployment", "outbound deployment"]
      instance_tags: { Name: "{{ item.name }}", type: ss, env: production}
      exact_count: "{{ count }}"
      count_tag: {  Name: "{{ item.name }}" }
      vpc_subnet_id: "{{ item.subnet }}" #MANUALLY HARDCODED
      wait: yes
    register: ec2
    with_items:
      - region: us-west-1
        name: ss12
        ami: ami-b33dccf7  # MANUALLY HARDCODED
        subnet: subnet-35a22550  # MANUALLY HARDCODED
      - region: ap-northeast-1
        name: ss21
        ami: ami-9e5cff9e  # MANUALLY HARDCODED
        subnet: subnet-88c47dff  # MANUALLY HARDCODED
      - region: eu-west-1
        name: ss32
        ami: ami-7c4b0a0b  # MANUALLY HARDCODED
        subnet: subnet-23f59554  # MANUALLY HARDCODED

While hardcoding ami/subnet works, can you think of a solution for me to avoid this hardcoding of the ami/subnet? I tried messing with set_fact to no avail as I couldn't get it to become a dictionary of "region to ami" value mappings

Answer 1

Keep in mind that Ansible is a "plugable" system, so it's really easy to customize it for your self. Sometimes it's even easier and faster than trying to find a workaround by using "native" modules.

In your case you can easily write your own custom lookup_plugin that would search for a correct subnet.

For example:

1. Create a folder called lookup_plugins in your main folder.
2. Create a file (if you do not have one) called ansible.cfg

[defaults]
lookup_plugins = lookup_plugins

Create a file in lookup_plugins called subnets.py

import boto.vpc
class LookupModule(object):
    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir
        self.plugin_name = 'subnets'
    def run(self, regions, variable=None, **kwargs):
        if not isinstance(regions, list):
            regions = [regions]
        for region in regions:
            return [boto.vpc.connect_to_region(region).get_all_subnets()[0].id]

The above simple code would look for a subnet in a given region. Of course you can customize it however you want.

Then in your playbook reference this plugin to find correct subnet:

Example:

- hosts: localhost
  gather_facts: no
  tasks:
    - name: Start instance
      debug: msg="Starting instance {{ item.ami }} in {{ item.region }} in {{ item.subnet }}"
      with_items:
        - region: us-west-1
          name: ss12
          ami: ami-b33dccf7  
          subnet: "{{ lookup('subnets', 'us-west-1') }}"
        - region: ap-northeast-1
          name: ss21
          ami: ami-9e5cff9e  
          subnet: "{{ lookup('subnets', 'ap-northeast-1') }}"
        - region: eu-west-1
          name: ss32
          ami: ami-7c4b0a0b 
          subnet: "{{ lookup('subnets', 'ap-northeast-1') }}"

In your case you would probably need to reference the correct AMI and associated Region .

Answer 2

if you still want to do this without help from another module you can calculate the modulo '%' of the servers and the subnets length:

"{{subnets[item.0 | int % subnets | length | int].aws_ec2_subnets}}"

example code

vars:

subnets:
  - {zone: "us-east-1a", aws_ec2_subnets: 'subnet-123'}
  - {zone: "us-east-1b", aws_ec2_subnets: 'subnet-456'}
  - {zone: "us-east-1d", aws_ec2_subnets: 'subnet-789'}

server_list:
  - server1
  - server2
  - server3

task:

- name: Create new ec2 instance
  ec2:
    profile: "{{aws_profile}}"
    key_name: "{{aws_key_name}}"
    group_id: "{{aws_security_group}}"
    instance_type: "{{aws_instance_type}}"
    image: "{{aws_ami}}"
    region: "{{region}}"
    exact_count: "1"
    #instance_profile_name: none
    wait: yes
    wait_timeout: 500
    volumes: "{{volumes}}"
    monitoring: no
    vpc_subnet_id: "{{subnets[item.0 | int % subnets | length | int].aws_ec2_subnets}}"
    assign_public_ip: no
    tenancy: default
    termination_protection: yes
    instance_tags:
      App: "{{app_name}}"
      Environment: "{{environment_type}}"
      Platform: "{{platform_name}}"
      Name: "{{item.1}}"
    count_tag:
      App: "{{app_name}}"
      Environment: "{{environment_type}}"
      Platform: "{{platform_name}}"
      Name: "{{item.1}}"
  register: ec2_new_instance
  with_indexed_items:
    - "{{server_list}}"

Answer 3

This is how I do it. The launchNodes.yml is just a simple ec2 with some taging

- debug:
    msg: "launching {{ nodeCount }} nodes in these subnets {{ec2SubnetIds}}"

- name: clear finalSubnetList
  set_fact:
    finalSubnetList: []

- name: build final list
  set_fact:
    finalSubnetList: "{{ finalSubnetList }} + [ '{{ ec2SubnetIds[ ( ec2subnet|int % ec2SubnetIds|length)|int ] }}' ]"
  with_sequence: count={{nodeCount}}  
  loop_control:
    loop_var: ec2subnet

- debug:
    msg: "finalSubnetList {{finalSubnetList}} "

- include_tasks: launchNodes.yml
      ec2SubnetId="{{ finalSubnetList[index|int -1] }}"
      nodeCount=1
  with_sequence: count="{{ finalSubnetList|length }}"
  loop_control:
    loop_var: index