I have some Terraform code which deploys AKS behind an Application Gateway along with other infrastructure unrelated to the issue. The code is based on https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-aks-applicationgateway-ingress. I can provide the actual files if needed.
All the infrastructure is being deployed successfully on initial creation, however when I configure the AGIC pods and deploy an application such as Nginx, it creates resources such as routing rules on the Application Gateway that is unknown to Terraform so if I run a Terraform deployment, it will revert back to what is in code (regardless if I made any changes or not). This causes the routing to my application through the ingress controller to return a 502 until it is redeployed.
How can I work around this?
> terraform apply -var-file=".\env\DEV.tfvars"
Acquiring state lock. This may take a few moments...
azurerm_resource_group.k8snetworkingrg: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG]
azurerm_resource_group.k8srg: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sRG]
azurerm_resource_group.acrrg: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sCRRG]
azurerm_resource_group.k8ssupportrg: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sSupportRG]
azurerm_container_registry.acr: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sCRRG/providers/Microsoft.ContainerRegistry/registries/DEVK8sCR]
azurerm_public_ip.publicip: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/publicIPAddresses/DEV-K8sPublicIP]
azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/virtualNetworks/DEV-K8sVNET]
azurerm_log_analytics_workspace.log: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourcegroups/DEV-k8ssupportrg/providers/microsoft.operationalinsights/workspaces/DEV-k8sloganalyticsworkspace]
data.azurerm_subnet.vnet_subnet1: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/virtualNetworks/DEV-K8sVNET/subnets/DevTestSubnet1]
azurerm_virtual_network_peering.vnet_peer: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/virtualNetworks/DEV-K8sVNET/virtualNetworkPeerings/DevTest-K8stoDevTest-Peering]
data.azurerm_subnet.vnet_subnet_ag: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/virtualNetworks/DEV-K8sVNET/subnets/DevTestAppGatewaySubnet]
azurerm_application_gateway.appgateway: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway]
azurerm_log_analytics_solution.log: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourcegroups/DEV-K8sSupportRG/providers/Microsoft.OperationsManagement/solutions/Containers(DEV-K8sLogAnalyticsWorkspace)]
azurerm_kubernetes_cluster.aks: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourcegroups/DEV-K8sRG/providers/Microsoft.ContainerService/managedClusters/DEV-K8sAKS]
azurerm_role_assignment.aks_sp_container_registry: Refreshing state... [id=/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sCRRG/providers/Microsoft.ContainerRegistry/registries/DEVK8sCR/providers/Microsoft.Authorization/roleAssignments/b8cd7474-fd5a-b4b0-3715-0c15e368c4fc]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# azurerm_application_gateway.appgateway will be updated in-place
~ resource "azurerm_application_gateway" "appgateway" {
enable_http2 = false
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway"
location = "northcentralus"
name = "DEV-K8sAppGateway"
resource_group_name = "DEV-K8sNetworkingRG"
~ tags = {
"Application" = "Kubernetes"
"Environment" = "DevTest"
"Owner" = "IT"
- "ingress-for-aks-cluster-id" = "/subscriptions/<Subscription_ID>/resourcegroups/DEV-k8srg/providers/Microsoft.ContainerService/managedClusters/DEV-k8saks" -> null
- "last-updated-by-k8s-ingress" = "2020-08-28 16:38:26.637125695 +0000 UTC m=+67.233012328" -> null
- "managed-by-k8s-ingress" = "1.2.0/32ea3149/2020-06-25-20:09T+0000" -> null
}
zones = []
~ backend_address_pool {
fqdns = []
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendAddressPools/defaultaddresspool"
ip_addresses = []
~ name = "defaultaddresspool" -> "DEV-K8sBackendAddressPool"
}
- backend_address_pool {
- fqdns = [] -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendAddressPools/pool-default-nginx-80-bp-80" -> null
- ip_addresses = [
- "10.11.0.23",
] -> null
- name = "pool-default-nginx-80-bp-80" -> null
}
~ backend_http_settings {
cookie_based_affinity = "Disabled"
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendHttpSettingsCollection/bp-default-nginx-80-80-nginx"
~ name = "bp-default-nginx-80-80-nginx" -> "DEV-K8sBackendHTTPSetting"
pick_host_name_from_backend_address = false
port = 80
probe_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/probes/pb-default-nginx-80-nginx"
- probe_name = "pb-default-nginx-80-nginx" -> null
protocol = "Http"
~ request_timeout = 30 -> 1
trusted_root_certificate_names = []
}
- backend_http_settings {
- cookie_based_affinity = "Disabled" -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendHttpSettingsCollection/defaulthttpsetting" -> null
- name = "defaulthttpsetting" -> null
- pick_host_name_from_backend_address = false -> null
- port = 80 -> null
- probe_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/probes/defaultprobe-Http" -> null
- probe_name = "defaultprobe-Http" -> null
- protocol = "Http" -> null
- request_timeout = 30 -> null
- trusted_root_certificate_names = [] -> null
}
frontend_ip_configuration {
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/frontendIPConfigurations/DEV-K8sFrontendIPConfig"
name = "DEV-K8sFrontendIPConfig"
private_ip_address_allocation = "Dynamic"
public_ip_address_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/publicIPAddresses/DEV-K8sPublicIP"
}
frontend_port {
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/frontendPorts/httpPort"
name = "httpPort"
port = 80
}
+ frontend_port {
+ id = (known after apply)
+ name = "httpsPort"
+ port = 443
}
gateway_ip_configuration {
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/gatewayIPConfigurations/DEV-K8sAppGatewayIpConfig"
name = "DEV-K8sAppGatewayIpConfig"
subnet_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/virtualNetworks/DEV-K8sVNET/subnets/DevTestAppGatewaySubnet"
}
~ http_listener {
frontend_ip_configuration_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/frontendIPConfigurations/DEV-K8sFrontendIPConfig"
frontend_ip_configuration_name = "DEV-K8sFrontendIPConfig"
frontend_port_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/frontendPorts/httpPort"
frontend_port_name = "httpPort"
host_names = []
id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/httpListeners/fl-e1903c8aa3446b7b3207aec6d6ecba8a"
~ name = "fl-e1903c8aa3446b7b3207aec6d6ecba8a" -> "DEV-K8sListenerHTTP"
protocol = "Http"
require_sni = false
}
- probe {
- host = "localhost" -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/probes/defaultprobe-Http" -> null
- interval = 30 -> null
- minimum_servers = 0 -> null
- name = "defaultprobe-Http" -> null
- path = "/" -> null
- pick_host_name_from_backend_http_settings = false -> null
- protocol = "Http" -> null
- timeout = 30 -> null
- unhealthy_threshold = 3 -> null
- match {
- status_code = [] -> null
}
}
- probe {
- host = "localhost" -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/probes/defaultprobe-Https" -> null
- interval = 30 -> null
- minimum_servers = 0 -> null
- name = "defaultprobe-Https" -> null
- path = "/" -> null
- pick_host_name_from_backend_http_settings = false -> null
- protocol = "Https" -> null
- timeout = 30 -> null
- unhealthy_threshold = 3 -> null
- match {
- status_code = [] -> null
}
}
- probe {
- host = "localhost" -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/probes/pb-default-nginx-80-nginx" -> null
- interval = 30 -> null
- minimum_servers = 0 -> null
- name = "pb-default-nginx-80-nginx" -> null
- path = "/" -> null
- pick_host_name_from_backend_http_settings = false -> null
- protocol = "Http" -> null
- timeout = 30 -> null
- unhealthy_threshold = 3 -> null
- match {
- status_code = [] -> null
}
}
- request_routing_rule {
- backend_address_pool_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendAddressPools/pool-default-nginx-80-bp-80" -> null
- backend_address_pool_name = "pool-default-nginx-80-bp-80" -> null
- backend_http_settings_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/backendHttpSettingsCollection/bp-default-nginx-80-80-nginx" -> null
- backend_http_settings_name = "bp-default-nginx-80-80-nginx" -> null
- http_listener_id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/httpListeners/fl-e1903c8aa3446b7b3207aec6d6ecba8a" -> null
- http_listener_name = "fl-e1903c8aa3446b7b3207aec6d6ecba8a" -> null
- id = "/subscriptions/<Subscription_ID>/resourceGroups/DEV-K8sNetworkingRG/providers/Microsoft.Network/applicationGateways/DEV-K8sAppGateway/requestRoutingRules/rr-e1903c8aa3446b7b3207aec6d6ecba8a" -> null
- name = "rr-e1903c8aa3446b7b3207aec6d6ecba8a" -> null
- rule_type = "Basic" -> null
}
+ request_routing_rule {
+ backend_address_pool_id = (known after apply)
+ backend_address_pool_name = "DEV-K8sBackendAddressPool"
+ backend_http_settings_id = (known after apply)
+ backend_http_settings_name = "DEV-K8sBackendHTTPSetting"
+ http_listener_id = (known after apply)
+ http_listener_name = "DEV-K8sListenerHTTP"
+ id = (known after apply)
+ name = "DEV-K8sRequestRoutingRuleHTTP"
+ redirect_configuration_id = (known after apply)
+ rewrite_rule_set_id = (known after apply)
+ rule_type = "Basic"
+ url_path_map_id = (known after apply)
}
sku {
capacity = 1
name = "WAF_v2"
tier = "WAF_v2"
}
ssl_policy {
cipher_suites = []
disabled_protocols = []
policy_name = "AppGwSslPolicy20150501"
policy_type = "Predefined"
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
