Terraform NAT Gateway forcing new resource every apply

Question

Thanks for your help in advance.(new to terraform)

Terraform is creating a new NAT gateway everytime during terraform apply. I am obviously missing something here,

resource "aws_eip" "tfeip"{
  vpc      = true
  depends_on = ["aws_internet_gateway.igw"]
  tags{
    Name = "${var.vpc_env}-tf-eip-nat-gw"
  }
}

# Create NAT Gateway
resource "aws_nat_gateway" "nat-gw" {
  allocation_id = "${aws_eip.tfeip.id}"
  subnet_id  = "${element(data.aws_subnet_ids.public-subnets.ids, 0)}"
  depends_on = ["aws_internet_gateway.igw"]
  tags{
    Name = "${var.vpc_env}-tf-nat-gateway"
  }
}

Terraform creates the new allocation ID and inturn the NAT gateway gets created everytime. Any idea what i am missing?

Here's the my plan output,

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_nat_gateway.nat-gw: Refreshing state...
data.aws_eip.eipid: Refreshing state...
aws_vpc.tfvpc: Refreshing state... [id=vpc-02f8c59a8497b4106]
aws_subnet.tfprivate[1]: Refreshing state... [id=subnet-08e2c6d4747f6825b]
aws_internet_gateway.igw: Refreshing state... [id=igw-0a2e4bac38e4daf38]
aws_subnet.tfprivate[0]: Refreshing state... [id=subnet-0b1e4b8c079eba45a]
aws_network_acl.tf-private-acl: Refreshing state... [id=acl-0c7e5678f7cc69c21]
aws_subnet.tfpublic[0]: Refreshing state... [id=subnet-055b2b384d0c0ce66]
aws_subnet.tfpublic[1]: Refreshing state... [id=subnet-048769cf43c37022b]
aws_eip.tfeip: Refreshing state... [id=eipalloc-0e5020f276b019d1a]
aws_route_table.tf-public-route: Refreshing state... [id=rtb-0c036c023e7949d24]
aws_nat_gateway.nat-gw: Refreshing state... [id=nat-0df993ed23f40559f]
aws_route_table_association.tf-public-route-association[0]: Refreshing state... [id=rtbassoc-0dbfff44cb44ef6e6]
aws_route_table_association.tf-public-route-association[1]: Refreshing state... [id=rtbassoc-05b2e9420129b6cad]
aws_route_table.tf-private-route: Refreshing state... [id=rtb-00d7850a8294282b5]
aws_route_table_association.tf-private-route-association[0]: Refreshing state... [id=rtbassoc-06d65de0b71461ac4]
aws_route_table_association.tf-private-route-association[1]: Refreshing state... [id=rtbassoc-04658d796d38247d7]
aws_vpc_endpoint.s3-vpce: Refreshing state... [id=vpce-0c48db5bd18368299]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_subnet_ids.private-subnets will be read during apply
  # (config refers to values not yet known)
 <= data "aws_subnet_ids" "private-subnets"  {
      + id     = (known after apply)
      + ids    = (known after apply)
      + tags   = {
          + "Type" = "Private"
        }
      + vpc_id = "vpc-02f8c59a8497b4106"
    }

  # data.aws_subnet_ids.public-subnets will be read during apply
  # (config refers to values not yet known)
 <= data "aws_subnet_ids" "public-subnets"  {
      + id     = (known after apply)
      + ids    = (known after apply)
      + tags   = {
          + "Type" = "Public"
        }
      + vpc_id = "vpc-02f8c59a8497b4106"
    }

  # aws_nat_gateway.nat-gw must be replaced
-/+ resource "aws_nat_gateway" "nat-gw" {
        allocation_id        = "eipalloc-0e5020f276b019d1a"
      ~ id                   = "nat-0df993ed23f40559f" -> (known after apply)
      ~ network_interface_id = "eni-09a821bxxxxx39297" -> (known after apply)
      ~ private_ip           = "10.38.0.21" -> (known after apply)
      ~ public_ip            = "3.225.5.235" -> (known after apply)
      ~ subnet_id            = "subnet-055b2b384d0c0ce66" -> (known after apply) # forces replacement
        tags                 = {
            "Name" = "dev-tf-nat-gateway"
        }
    }

  # aws_route_table.tf-private-route will be updated in-place
  ~ resource "aws_route_table" "tf-private-route" {
        id               = "rtb-00d7850a8294282b5"
        owner_id         = "xxxxxxxxxxxxx"
        propagating_vgws = []
      ~ route            = [
          - {
              - cidr_block                = "0.0.0.0/0"
              - egress_only_gateway_id    = ""
              - gateway_id                = ""
              - instance_id               = ""
              - ipv6_cidr_block           = ""
              - nat_gateway_id            = "nat-0df993ed23f40559f"
              - network_interface_id      = ""
              - transit_gateway_id        = ""
              - vpc_peering_connection_id = ""
            },
            {
                cidr_block                = "0.0.0.0/0"
                egress_only_gateway_id    = ""
                gateway_id                = ""
                instance_id               = ""
                ipv6_cidr_block           = ""
                nat_gateway_id            = (known after apply)
                network_interface_id      = ""
                transit_gateway_id        = ""
                vpc_peering_connection_id = ""
            },
        ]
        tags             = {
            "Name" = "dev-tf-private-rt-table"
        }
        vpc_id           = "vpc-02f8c59a8497b4106"
    }

  # aws_route_table_association.tf-private-route-association[0] must be replaced
-/+ resource "aws_route_table_association" "tf-private-route-association" {
      ~ id             = "rtbassoc-06d65de0b71461ac4" -> (known after apply)
        route_table_id = "rtb-00d7850a8294282b5"
      ~ subnet_id      = "subnet-08e2c6d4747f6825b" -> (known after apply) # forces replacement
    }

  # aws_route_table_association.tf-private-route-association[1] must be replaced
-/+ resource "aws_route_table_association" "tf-private-route-association" {
      ~ id             = "rtbassoc-04658d796d38247d7" -> (known after apply)
        route_table_id = "rtb-00d7850a8294282b5"
      ~ subnet_id      = "subnet-0b1e4b8c079eba45a" -> (known after apply) # forces replacement
    }

  # aws_route_table_association.tf-public-route-association[0] must be replaced
-/+ resource "aws_route_table_association" "tf-public-route-association" {
      ~ id             = "rtbassoc-0dbfff44cb44ef6e6" -> (known after apply)
        route_table_id = "rtb-0c036c023e7949d24"
      ~ subnet_id      = "subnet-048769cf43c37022b" -> (known after apply) # forces replacement
    }

  # aws_route_table_association.tf-public-route-association[1] must be replaced
-/+ resource "aws_route_table_association" "tf-public-route-association" {
      ~ id             = "rtbassoc-05b2e9420129b6cad" -> (known after apply)
        route_table_id = "rtb-0c036c023e7949d24"
      ~ subnet_id      = "subnet-055b2b384d0c0ce66" -> (known after apply) # forces replacement
    }

Plan: 5 to add, 1 to change, 5 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.```

Can you show the plan output when it wants to recreate the NAT gateway? — ydaetskcoR
@ydaetskcoR please find the below plan output attached in the question section — Karthik
It looks like it thinks the subnet ID has changed for where you want the NAT gateway. Are you saying that if you apply and then plan again it wants to recreate the next time as well and shows a different subnet ID for the NAT? — ydaetskcoR
Your plan output also includes some data resources like data "aws_subnet_ids" "private-subnets", which Terraform says must be refreshed during the apply step rather than the plan step, and I expect that's the root cause here. Could you share the configuration for those data blocks as part of your question too? — Martin Atkins

Karthik Karthik · Accepted Answer · 2019-08-09T16:03:41

@ydaetskcoR The issue is resolved after creating one gateway per AZ on all the public subnets and removing the old orphan gateway from the state file.

Terraform NAT Gateway forcing new resource every apply

1 Answers