AWX job template doesn't recognize an aws_ec2 inventory group, but ansible-playbook does - same yaml file for both

Question

I'm trying to run awx jobs against aws ec2 instances, but I keep getting the warning '[WARNING] Could not match supplied host pattern, ignoring: security_groups_webservers". If I run the same playbook from the command line, it works.

Versions:

    awx:        17.0.1 (same problem on 15.0.1)
    ansible:    2.9.17

ansible.cfg

    [defaults]
    inventory = /opt/ansible/inventory/aws_ec2.yml
    enable_plugins = aws_ec2
    host_key_checking = false
    private_key_file = [the key identified on the EC2 instance]
    remote_user = ubuntu

The job inventory /opt/ansible/inventory/aws_ec2.yml:

    plugin: aws_ec2
    regions:
      - us-east-2
    strict: false
    keyed_groups:
      - key: tags
        prefix: tag
      - key: placement.region
        prefix: aws_region
      - key: 'security_groups|json_query("[].group_name")'
        prefix: 'security_groups'
    hostnames:
      - ip-address
      - dns-name
      - tag:Name
      - private-ip-address

Notice how the below hosts are listed by their IP addresses, not the public dns names. This is because of the 'hostnames' section of the above inventory file 'aws_ec2.yml'. The hostname section tells ansible-inventory in a list of priorities, how to display the hosts.

ansible-inventory --graph

    @all:
    |--@aws_ec2:
    |  |--###.###.166.103
    |  |--###.###.76.94
    |  @aws_region_us_east_2:
    |  |--###.###.166.103
    |  |--###.###.76.94
    |  @security_groups_dbservers:
    |  |--###.###.166.103
    |  @security_groups_webservers:
    |  |--###.###.76.94
    ...

A simple ping playbook called 'ping.yml'

    ---
    - hosts:
        - security_groups_dbservers
        - security_groups_webservers
      tasks:
      - name: ping hosts
        ping:

Running the playbook from the command line:

ansible-playbook ping.yml

    PLAY [security_groups_dbservers,security_groups_webservers]****
    
    TASK [Gathering Facts] ****************************************
    ok: [###.###.76.94]
    ok: [###.###.166.103]
    
    TASK [ping hosts] *********************************************
    ok: [###.###.76.94]
    ok: [###.###.166.103]
    
    PLAY RECAP ****************************************************
    ###.###.166.103   : ok=2  changed=0   unreachable=0   failed=0   skipped=0   rescued=0   ignored=0
    ###.###.76.94     : ok=2  changed=0   unreachable=0   failed=0   skipped=0   rescued=0   ignored=0

AWX project detail:

    Name: ping-project
    Organization: Default
    Source Control Credential Type: Manual
    Project Base Path: /var/lib/awx/projects
    Playbook directory: lab1

Inventory:

    Name: aws-ec2-inventory
    Organization: Default

Inventory Sources: (synchronized)

    Name: aws-ec2-source
    Source: Amazon EC2
    Credential: aws-iam-user
    
    Overwrite: (checked)

Inventory Hosts:

    ec2-###-###-166-103.us-east-2.compute.amazonaws.com
    ec2-###-###-76-94.us-east-2.compute.amazonaws.com

Here's the first sign that something's not working. The hosts are listed by their dns names. If awx was using the same inventory file as ansible-inventory, they would show up as IP addresses.

Here are the AWX details (facts) for one of the hosts:

...
network_interfaces:
  - association:
    ...
    groups:
        group_name: dbservers
    ...
...
security_groups:
    group_name: dbservers
...

It's obviously not associating the 'security_groups_dbservers' group generated in the aws_ec2 plugin with the above security_groups.group_name: dbservers. So I guess I have to convert the group name in the playbook to speak in awx terminology. Question is, how?

Credentials:

    (AWS IAM user)
    
    Name: aws-iam-user
    Orgainzation: Default
    Credential Type: Amazon Web Services
    Access Key: [aws iam user's ACCESS key]
    Secret Key: [aws iam user's SECRET key]
    
    (AWS Ubuntu t2.micro instance)

    Name: ubuntu-machine-ssh
    Organization: Default
    Credential Type: Machine
    Username: ubuntu
    SSH Private Key: [the key identified on the EC2 instance]

AWX Job Template:

    Name: ping-hosts
    Job Type: Run
    Inventory: aws-ec2-inventory
    Project: ping-project
    Playbook: ping.yml
    Credentials: ubuntu-machine-ssh

Output:

    Identity added: /tmp/awx_288_j3vtwbvj/artifacts/288/ssh_key_data 
    (/tmp/awx_288_j3vtwbvj/artifacts/288/ssh_key_data)
    
    [WARNING]: Could not match supplied host pattern, ignoring:
    security_groups_dbservers
    [WARNING]: Could not match supplied host pattern, ignoring:
    security_groups_webservers
    
    PLAY [security_groups_dbservers,security_groups_webservers] ********************
    skipping: no hosts matched
    
    PLAY RECAP *********************************************************************

Debug output:

    Using /tmp/awx_288_j3vtwbvj/project/ansible.cfg as config file
    host_list declined parsing /tmp/awx_288_j3vtwbvj/tmpbc_m8p2g as it did 
    not pass its verify_file() method

The thing that jumps out at me here, is that it says the 'host_list' module declined parsing. But in ansible.cfg, the 'aws_ec2' plugin is enable, which should have put it in the list of 'inventories' to check along with the 'host_list' module. It's saying the 'aws_ec2' plugin wasn't even checked.

Any suggestions?

Answer 1

[SOLVED] awx apparently isn't importing the inventory file from the 'inventory' setting in ansible.cfg. The way I solved this was to copy the inventory file (aws_ec2.yml) contents into the 'SOURCES VARIABLES' text box of the project inventory source, resynchronized the inventory source and whalah! Every host now has the groupings I need, and can list the hosts by their groups in the job playbooks. I modified the ping job to list hosts by their security groups (i.e., 'security_groups_webservers','security_groups_dbservers'), and the awx job template worked. Just like the command line run!