Terraform NSG rules from multiple module outputs with for_each

Question

Looking for some guidance on the following problem I'm facing...

We're defining maps in our .tfvars which are referenced using for_each to deploy our modules. This has worked well for some time... However, we recently deployed a clustered environment. So, we now have two application servers, defined in .tfvars:

app_servers map in tfvars:

app_servers = {
  sr-app-1 = {
    size             = "Standard_E2s_v3"
    admin_username   = "azureuser"
    data_disks       = [64]
    zone_vm          = "1"
    zone_disk        = ["1"]
    os_disk_size     = 64
    web_server       = "sr-web-1"
  },
  sr-app-2 = {
    size             = "Standard_E2s_v3"
    admin_username   = "azureuser"
    data_disks       = [64]
    zone_vm          = "1"
    zone_disk        = ["1"]
    os_disk_size     = 64
    web_server       = "sr-web-2"
  }
}

db_servers map in tfvars

db_servers = {
  sr-fdb-1 = {
    size             = "Standard_D4s_v3"
    admin_user       = "azureuser"
    data_disks       = [64, 64, 128]
    zone_vm          = "1"
    zone_disk        = ["1"]
    os_disk_size     = 128
    app_server       = "sr-app-1"
    etl_server       = "sr-etl-1"
  }
  sr-sdb-1 = {
    size             = "Standard_D4s_v3"
    admin_user       = "azureuser"
    data_disks       = [64, 64, 128]
    zone_vm          = "1"
    zone_disk        = ["1"]
    os_disk_size     = 128
    app_server       = "sr-app-1"
    etl_server       = "sr-etl-1"
  }
}

We are populating the source_address_prefixes for our NSG rules (outside of the module, in main.tf) by using the outputs.tf from the underlying linux_vm module. Because we are using a for_each on the db_servers map (below), we can't access the app_servers map. So, we added a key to the db_servers map to identify the particular module run for each of the app_servers, (see above db_servers map):

outputs.tf (inside linux_vm module):

output "linux_vm_ip" {
  value = azurerm_network_interface.uks_network_interface.private_ip_address
}

output "linux_vm_nsg" {
  value = azurerm_network_security_group.uks_network_security_group.name
}

main.tf:

module "fico_db_vm" {
  for_each                = var.db_servers
  source                  = "../modules/compute/windows_vm"
  source_image_id         = var.db_image_id
  vm_name                 = each.key
  vm                      = each.value
  subnet_name             = "back-end-01"
  resource_group          = azurerm_resource_group.rg_uks_fico_db.name
  data_disks              = each.value["data_disks"]
  enable_management_locks = true
}

resource "azurerm_network_security_rule" "fico-db-sr-1433" {
  for_each                    = var.db_servers
  name                        = "nsr-${var.location}-${var.environment}-${var.directorate}-${var.business_unit}-sql"
  priority                    = 100
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "*"
  source_port_range           = "*"
  destination_port_range      = "1433"
  source_address_prefixes     = [module.fico_app_vm[each.value.app_server].linux_vm_ip, module.fico_etl_vm[each.value.etl_server].windows_vm_ip]
  destination_address_prefix  = "VirtualNetwork"
  resource_group_name         = azurerm_resource_group.rg_uks_fico_db.name
  network_security_group_name = module.fico_db_vm[each.key].windows_vm_nsg
}

This works well in our SBOX/DEV environments which are all single instances - However, we have now deployed a clustered application in to our TEST environment. I can't figure out a nice way to add the secondary app servers IP to the above NSG rule.

Initially I tried to add a new key (app_server2) to the db_servers map, so that can be reference in the same way: source_address_prefixes = [module.fico_app_vm[each.value.app_server].linux_vm_ip, module.fico_app_vm[each.value.app_server2].linux_vm_ip, module.fico_etl_vm[each.value.etl_server].windows_vm_ip]

It works as expected in TEST, but this isn't feasible as the Terraform plan complains when running the same code in our existing SBOX/DEV environments as app_server2 is not defined...

I have tried to add app_server2 to the SBOX/DEV tfvars as "null" or just an empty value, but that too complains as it appears that maps cannot be empty/null?

Maybe I am too far down the rabbit hole to see an obvious solution, any advice/guidance would be appreciated.

Answer 1

I ended up getting around this particular problem by removing my references to other maps, in my maps...

Instead, I added a few local variables in the main.tf

locals {
  app_vm_ips = [
    for key, val in module.fico_app_vm :
    module.fico_app_vm[key].linux_vm_ip
  ]
  web_vm_ips = [
    for key, val in module.fico_web_vm :
    module.fico_web_vm[key].linux_vm_ip
  ]
  etl_vm_ips = [
    for key, val in module.fico_etl_vm :
    module.fico_etl_vm[key].windows_vm_ip
  ]
}

I then used these local variables in the azurerm_network_security_group_rule resource like so:

resource "azurerm_network_security_rule" "fico-db-sr-1433" {
  for_each                    = var.db_servers
  name                        = "nsr-${var.location}-${var.environment}-${var.directorate}-${var.business_unit}-sql"
  priority                    = 100
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "*"
  source_port_range           = "*"
  destination_port_range      = "1433"
  source_address_prefixes     = concat(local.app_vm_ips, local.etl_vm_ips)
  destination_address_prefix  = "VirtualNetwork"
  resource_group_name         = azurerm_resource_group.rg_uks_fico_db.name
  network_security_group_name = module.fico_db_vm[each.key].windows_vm_nsg
}