Ansible: (unexpected) hostvars items always overwritten by values for last host in group during task loop

Question

I am attempting to write an Ansible playbook that runs a handful of tasks to configure Cisco IOS routers. Some of these tasks need to loop over list variables that are defined at the host variables file level. For example, given one or more interfaces, configure x on that interface. Or, given one or more fvrfs, configure nameservers for each fvrf. The number of interfaces and fvrfs are dynamic. There are multiple tasks that have dynamic list values like this within the playbook role.

The problem that I am running into is that host variables with unique values for each router are always being set to the values defined for the last router in the group. This is happening for variables that are a string and variables that are a list of strings. In other words, the hostvars of the routers preceding the last router in the inventory group are always being overwritten by the hostvars defined for the last router.

Ansible Runtime:

$ ansible --version
ansible 2.7.0
  config file = /opt/ansible/ansible.cfg
  configured module search path = [u'/home/<redacted>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/<redacted>/mypython/lib/python2.7/site-packages/ansible
  executable location = /home/<redacted>/mypython/bin/ansible
  python version = 2.7.15 (default, Oct 22 2018, 15:22:25) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
(A) (mypython) <redacted>@<redacted_hostname> /opt/ansible
$ ansible-playbook --version
ansible-playbook 2.7.0
  config file = /opt/ansible/ansible.cfg
  configured module search path = [u'/home/<redacted>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/<redacted>/mypython/lib/python2.7/site-packages/ansible
  executable location = /home/<redacted>/mypython/bin/ansible-playbook
  python version = 2.7.15 (default, Oct 22 2018, 15:22:25) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]

Directory structure:

+-- ansible.cfg
+-- inventory
|   +-- lab-g2
|   |   +-- group_vars
|   |   |   +-- lab-g2-crs-2900
|   |   |       +-- host_vars
|   |   |       |   +-- 10.74.0.71.yml
|   |   |       |   +-- 10.74.0.73.yml
|   |   |       +-- vars
|   |   +-- inventory
+-- library
+-- playbooks
|   +-- roles -> /opt/ansible/roles
|   +-- set-nameservers.yml
+-- README.md
+-- roles
|    +-- set-nameservers
    |   +-- tasks
    |       +-- main.yml

playbook.yml:

---

- name: CONFIGURE NAMESERVERS ON ROUTER
  hosts: all
  gather_facts: no
  connection: network_cli

  roles:
    - set-nameservers

Inventory file:

[lab-g2-crs-2900]
10.74.0.71
10.74.0.73

[all:children]
lab-g2-crs-2900

Group variables file:

---

ansible_connection: network_cli
ansible_network_os: ios

Host variable files:

10.74.0.71.yml:

fvrf: ["WAN1", "WAN2"]
umbrella_out: ["GigabitEthernet0/0"]

10.74.0.73.yml:

fvrf: ["WAN3", "WAN4"]
umbrella_out: ["GigabitEthernet0/1"]

roles/set-nameservers/tasks/main.yml

---
- name: CONFIGURE NAMESERVERS
  ios_config:
    lines:
      - "ip name-server vrf {{ item }} 208.67.220.220 208.67.222.222"
  with_items: "{{ fvrf }}"

- name: DEBUG
  debug:
    msg: "fvrf name is {{ item }}"
  with_items: "{{ fvrf }}"

- name: CONFIGURE UMBRELLA OUTBOUND INTERFACE
  ios_config:
    lines:
      - "description Outbound umbrella interface"
    parents: interface {{ item }}
  with_items: "{{ umbrella_out }}"

- name: DEBUG
  debug:
    msg: "Outbound Umbrella interface is {{ item }}"
  with_items: "{{ umbrella_out }}"

EXPECTED RESULT

PLAY [CONFIGURE NAMESERVERS ON ROUTER] ***************************************************************************************************************************************

TASK [set-nameservers : CONFIGURE NAMESERVERS] *******************************************************************************************************************************
changed: [10.74.0.73] => (item=WAN3)
changed: [10.74.0.71] => (item=WAN1)
changed: [10.74.0.73] => (item=WAN4)
changed: [10.74.0.71] => (item=WAN2)

TASK [set-nameservers : DEBUG] ***********************************************************************************************************************************************
ok: [10.74.0.71] => (item=WAN3) => {
    "msg": "fvrf name is WAN1"
}
ok: [10.74.0.71] => (item=WAN4) => {
    "msg": "fvrf name is WAN2"
}
ok: [10.74.0.73] => (item=WAN3) => {
    "msg": "fvrf name is WAN3"
}
ok: [10.74.0.73] => (item=WAN4) => {
    "msg": "fvrf name is WAN4"
}

TASK [set-nameservers : CONFIGURE UMBRELLA OUTBOUND INTERFACE] ***************************************************************************************************************
changed: [10.74.0.73] => (item=GigabitEthernet0/0)
changed: [10.74.0.71] => (item=GigabitEthernet0/1)

TASK [set-nameservers : DEBUG] ***********************************************************************************************************************************************
ok: [10.74.0.71] => (item=GigabitEthernet0/1) => {
    "msg": "Outbound Umbrella interface is GigabitEthernet0/0"
}
ok: [10.74.0.73] => (item=GigabitEthernet0/1) => {
    "msg": "Outbound Umbrella interface is GigabitEthernet0/1"
}

PLAY RECAP *******************************************************************************************************************************************************************
10.74.0.71                : ok=4    changed=2    unreachable=0    failed=0
10.74.0.73                : ok=4    changed=2    unreachable=0    failed=0

ACTUAL RESULT

PLAY [CONFIGURE NAMESERVERS ON ROUTER] ***************************************************************************************************************************************

TASK [set-nameservers : CONFIGURE NAMESERVERS] *******************************************************************************************************************************
changed: [10.74.0.73] => (item=WAN3)
changed: [10.74.0.71] => (item=WAN3)
changed: [10.74.0.73] => (item=WAN4)
changed: [10.74.0.71] => (item=WAN4)

TASK [set-nameservers : DEBUG] ***********************************************************************************************************************************************
ok: [10.74.0.71] => (item=WAN3) => {
    "msg": "fvrf name is WAN3"
}
ok: [10.74.0.71] => (item=WAN4) => {
    "msg": "fvrf name is WAN4"
}
ok: [10.74.0.73] => (item=WAN3) => {
    "msg": "fvrf name is WAN3"
}
ok: [10.74.0.73] => (item=WAN4) => {
    "msg": "fvrf name is WAN4"
}

TASK [set-nameservers : CONFIGURE UMBRELLA OUTBOUND INTERFACE] ***************************************************************************************************************
changed: [10.74.0.73] => (item=GigabitEthernet0/1)
changed: [10.74.0.71] => (item=GigabitEthernet0/1)

TASK [set-nameservers : DEBUG] ***********************************************************************************************************************************************
ok: [10.74.0.71] => (item=GigabitEthernet0/1) => {
    "msg": "Outbound Umbrella interface is GigabitEthernet0/1"
}
ok: [10.74.0.73] => (item=GigabitEthernet0/1) => {
    "msg": "Outbound Umbrella interface is GigabitEthernet0/1"
}

PLAY RECAP *******************************************************************************************************************************************************************
10.74.0.71                : ok=4    changed=2    unreachable=0    failed=0
10.74.0.73                : ok=4    changed=2    unreachable=0    failed=0

As you can see from the output result, the variables from the host file for 10.74.0.73 are being used for both hosts, even though 10.74.0.71 has its own unique variable values defined in a separate file. In a separate playbook with the same structure and 16 routers, it was showing the same behavior... using the hostvars of the last router in the group for all 16 routers (yikes!)

I've been searching the web for a few hours and have looked through the Ansible docs and a lot of discussion here pertaining to loops, variables, and variable precedence. I haven't figured out what the issue is. I think the most likely culprit is that I am misunderstanding how the with_items operation works, but I don't know how to modify the tasks to ensure the desired result where each host has unique variable lists. Could this possibly be bug behavior?

Any assistance with this issue is greatly appreciated!

Hi Geoff, welcome to SO! What's going on with ansible 2.7.0.dev0? Did you build ansible from git? Are you able to reproduce this behavior with a released version? — mdaniel
Matt, I noticed that issue as well when I was troubleshooting and went through the process of setting up a fresh Python virtualenv with the stable release of 2.7. I'm getting the same results, however. I updated the runtime information in my original post. All of the result output was generated with the stable release version, I just neglected to update the runtime information with the new version. — liltechdude

mdaniel mdaniel · Accepted Answer · 2018-10-23T04:44:19

Thanks for posting the directory output, that was the key. It's not the with_items that's causing you the problems, it's the nested host_vars within group_vars.

If you pull the host_vars up out of the group_vars directory, it will start to behave sanely again. I haven't dug into the assignment rules to find out exactly at which point it is being superseded, but the tl;dr is that the top bucket is not host specific, even though the bottom bucket is, thus:

hostvars["10.74.0.71"] = # the correct thing 
vars["lab-g2-crs-2900"] = {}
for h in ["10.74.0.71", ...etc...]:
   vars["lab-g2-crs-2900"].update(hostvars[h])
# and now the value in the group_vars "masks off" the host-specific one
# because they appear to be applied in reverse-depth-first order

You can confirm for yourself without running the entire playbook just using ansible-inventory --list and looking at the hostvars in _meta

Ansible: (unexpected) hostvars items always overwritten by values for last host in group during task loop

1 Answers