issues with creating cross region read replica cluster for RDS aurora using terraform

1
votes

UPDATE: issue #2 is actually not terraform issue..issue was with AWS..i fixed that..code works fine..only thing left now is issue #1 with the AZs issue

i have a RDS aurora cluster with 1 instance in it in us-east-1 and want to create a cross region read replica RDS aurora cluster with 1 instance in it in us-west-2

there are like 2 issues am having when trying to create a cross region read replica from an RDS aurora cluster using terrform

here is my main.tf file

provider "aws" {
  region = "${var.aws_region}"

}

resource "aws_db_subnet_group" "rds-aurora" {
  name       = "${var.name}-${var.aws_region}"
  subnet_ids = ["${split(",",lookup(var.subnet_ids, "${var.aws_region}"))}"]
  tags {
    Name            = "${var.name}-${var.env}-${var.aws_region}"
  }
}

resource "aws_rds_cluster_parameter_group" "rds-aurora-cluster-pg" {
  name        = "${var.name}-cluster-pg-${var.aws_region}"
  family      = "aurora5.6"
  description = "${var.name} cluster parameter group"
  parameter {
    name  = "secure_auth"
    value = "1"
  }
  parameter {
    name  = "binlog_format"
    value = "MIXED"
    apply_method = "pending-reboot"
  }
  tags {
    Name            = "${var.name}-${var.env}-${var.aws_region}"
  }
  lifecycle {
      create_before_destroy = true
  }  
}

resource "aws_db_parameter_group" "rds-aurora-pg" {
  name   = "${var.name}-pg-${var.aws_region}"
  family = "aurora5.6"
  description = "${var.name} parameter group"
  parameter {
    name  = "secure_auth"
    value = "1"
  }
  tags {
    Name            = "${var.name}-${var.env}-${var.aws_region}"
  }
  lifecycle {
      create_before_destroy = true
  }
}

resource "aws_rds_cluster" "rds-aurora" {
  cluster_identifier      = "${var.name}"
  availability_zones      = ["${lookup(var.availability_zones, var.aws_region)}"]  
  skip_final_snapshot = "true"
  backup_retention_period = "${var.rds_backup}"
  preferred_backup_window = "09:00-09:30"
  preferred_maintenance_window = "mon:06:00-mon:06:30"
  apply_immediately = "true"
  engine_mode       = "provisioned"
  deletion_protection = "false"
  vpc_security_group_ids = ["${split(",",lookup(var.security_groups, var.aws_region))}"]
  storage_encrypted = "true"
  db_subnet_group_name = "${aws_db_subnet_group.rds-aurora.name}"
  db_cluster_parameter_group_name = "${aws_rds_cluster_parameter_group.rds-aurora-cluster-pg.name}"
  kms_key_id = "${var.kms_key_id}"
  replication_source_identifier = "${var.replication_source_identifier}${var.name}"
  source_region = "${var.source_region}"
  tags {
    Name            = "${var.name}-${var.env}-${var.aws_region}"
  }
}

resource "aws_rds_cluster_instance" "rds-aurora" {
  count              = "${var.rds_count}"
  identifier         = "${var.name}-${count.index}"
  engine = "aurora"
  db_subnet_group_name = "${aws_db_subnet_group.rds-aurora.name}"
  db_parameter_group_name = "${aws_db_parameter_group.rds-aurora-pg.name}"
  instance_class     = "${var.instance_class}"
  cluster_identifier = "${aws_rds_cluster.rds-aurora.cluster_identifier}"
  tags {
    Name            = "${var.name}-${var.env}-${var.aws_region}"
  }
}

and here are the issues am facing

1. first issue is am getting this error

aws_rds_cluster.rds-aurora: error creating RDS cluster: InvalidVPCNetworkStateFault: Availability zone '[us-west-2a,us-west-2b,us-west-2c]' is unavailable in this region, please choose another zone set.

and those availability zones do exist and i believe i have entered the values in the proper way here is how terraform got the values for the availabiity zones

...
availability_zones.2635104823:     "" => "us-west-2a,us-west-2b,us-west-2c"
...

which is correct format and the AZs exist like i said..i have created RDS from scratch and same format works but now trying to create a cross region read replica is only when i get the error that the AZs dont exist

now i was able to move forward to next issue by adding just 1 AZ to get past this stage..but the issue still exists and i will like to fix that as well

n ow next is issue #2 below...the main issue

2. now the major error is the bug i believe exist with creating cross region read replica for RDS aurora with terraform

the terrform apply gets stuck when creating the rds cluster and it never completes...it runs here for like more than 2 hours before a timeout kicks in..and one thing i noticed is when i check the AWS dashboard i can see the RDS cluster is created but there is no RDS instance under it but terrform keeps runing the creation of the RDS cluster forever

aws_rds_cluster.rds-aurora: Still creating... (10s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (20s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (30s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (40s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (50s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (1m0s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (1m10s elapsed)
...
...
...
aws_rds_cluster.rds-aurora: Still creating... (1h59m40s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (1h59m50s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (2h0m0s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (2h0m10s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (2h0m20s elapsed)
aws_rds_cluster.rds-aurora: Still creating... (2h0m30s elapsed)

so i think there is a bug where terrafrom does not know when the RDS cluster is created so it can create the RDS instance under it as specified in the main.tf file above

anyone willing to help with these 2 issues?

2
terraformamazon-rdsterraform-provider-awsamazon-aurora
It's still not clear if you are trying to create a read replica cluster or if you are trying to create just a read replica instance. If you are attempting to create a read replica cluster then using replication_source_identifier makes sense. I can't see your variable definitions for aws_region or source_region but are they in different regions and do you have the correct AZ's for each region also defined in your variables? - Michael Quale
did you see this line under aws_rds_cluster resource replication_source_identifier = "${var.replication_source_identifier}${var.name}"? that is where i am referencing a remote rds cluster to create the read replica from - uberrebu
I have the exact same issue as num 2. Figured it out? - jetru
@jetru can you please share your solution? - uberrebu
Don't have a solution yet. :( - jetru

2 Answers

0
votes

Ok, so you are creating read replica cluster instead of read replica instance, are you specifying a db instance or a cluster as source? You should be setting the source as a db_instance in the cluster, not the source cluster. Ideally a read replica if you are using a multi AZ HA source.

One thing to keep in mind, since I dont know your environment and have to shoot from the hip, if you using a single host cluster. Don't bother using a cluster. Just use a single db instance. It's simpler and relatively the same

0
votes

What I am saying is that is difficult to see what you are doing based on the TF code you have provided. Where is the source cluster TF code and your variable definitions?