Can't execute neither user_data script, nor remote-exec with connection block while launching ec2 instance with terraform cloud [aws-provider]

Question

I have created a aws infrastructure with network acls, security group, subnets, etc [code attached at the bottom]. in the free tier. I have also established ssh connection with my ec2 instance and I can also download manually packages when logged to the instance.

However, since I want to fully utilize Terraform, I would like to pre-install some stuff while Terraform creates the instance.

The commands I want to execute are quite simple (install jdk, python, docker),

user_data= <<-EOF
#! /bin/bash
    echo "Installing modules..."
    sudo apt-get update
    sudo apt-get install -y openjdk-8-jdk
    sudo apt install -y python2.7 python-pip
    sudo apt install -y docker.io
    sudo systemctl start docker
    sudo systemctl enable docker
    pip install setuptools
    echo "Modules installed via Terraform"
EOF

My first approach was to utilize user_data parameter. Even though ec2 instance has access to the internet, none of the modules specified have been installed. Then I utilized the remote-exec block along with the connection block provided by terraform. But as many of us experienced before, terraform can't establish a successful connection to host, giving back the following messages,

remote-exec block

connection {
  type        = "ssh"
  host        = aws_eip.prod_server_public_ip.public_ip //Error: host for provisioner cannot be empty -> https://github.com/hashicorp/terraform-provider-aws/issues/10977
  user        = "ubuntu"
  private_key = "${chomp(tls_private_key.ssh_key_prod.private_key_pem)}"
  timeout     = "1m"
}

provisioner "remote-exec" {
  inline = [
    "echo 'Installing modules...'",
    "sudo apt-get update",
    "sudo apt-get install -y openjdk-8-jdk",
    "sudo apt install -y python2.7 python-pip",
    "sudo apt install -y docker.io",
    "sudo systemctl start docker",
    "sudo systemctl enable docker",
    "pip install setuptools",
    "echo 'Modules installed via Terraform'"
  ]
  on_failure = fail
}

message of i/o timeout

Connecting to remote host via SSH...
module.virtual_machines.null_resource.install_modules (remote-exec):   Host: 3.137.111.207
module.virtual_machines.null_resource.install_modules (remote-exec):   User: ubuntu
module.virtual_machines.null_resource.install_modules (remote-exec):   Password: false
module.virtual_machines.null_resource.install_modules (remote-exec):   Private key: true
module.virtual_machines.null_resource.install_modules (remote-exec):   Certificate: false
module.virtual_machines.null_resource.install_modules (remote-exec):   SSH Agent: false
module.virtual_machines.null_resource.install_modules (remote-exec):   Checking Host Key: false
module.virtual_machines.null_resource.install_modules (remote-exec):   Target Platform: unix

timeout - last error: dial tcp 52.15.178.40:22: i/o timeout

One root of the problem that I could think of, is that I allow only 2 specific ip addresses to pass form the inbound routing of security group. So when terraform tries to connect it does so from an unknown ip to the security group. If that's the case, which is the IP address that would allow terraform to connect to my vm and pre-install packages?

Terraform code for the infrastructure.

Answer 1

I run your code in my sandbox env, and the remote-exec works. I had to make some changes for it to work and even to run your code (region, ami, security groups, ...). So you can have a look at the modified code and take it from there. But the code below works for me without any issues.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}



variable "prefix" {
    default="my"
}

# Create virtual private cloud (vpc)
resource "aws_vpc" "vpc_prod" {
  cidr_block = "10.0.0.0/16" #or 10.0.0.0/16
  enable_dns_hostnames = true
  enable_dns_support = true

  tags = {
      Name = "production-private-cloud"
  }
}

# Assign gateway to vp
resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.vpc_prod.id

  tags = {
      Name = "production-igw"
  }
}

# ---------------------------------------- Step 1: Create two subnets ----------------------------------------
data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "subnet_prod" {
  vpc_id            = aws_vpc.vpc_prod.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a" #data.aws_availability_zones.available.names[0]
  depends_on        = [aws_internet_gateway.gw]

  map_public_ip_on_launch = true

  tags = {
      Name = "main-public-1"
  }
}

resource "aws_subnet" "subnet_prod_id2" {
  vpc_id            = aws_vpc.vpc_prod.id
  cidr_block        = "10.0.2.0/24" //a second subnet can't use the same cidr block as the first subnet
  availability_zone = "us-east-1b" #data.aws_availability_zones.available.names[1]
  depends_on        = [aws_internet_gateway.gw]

  tags = {
        Name = "main-public-2"
    }
}

# ---------------------------------------- Step 2: Create ACL network/ rules ----------------------------------------
resource "aws_network_acl" "production_acl_network" {
  vpc_id = aws_vpc.vpc_prod.id
  subnet_ids = [aws_subnet.subnet_prod.id, aws_subnet.subnet_prod_id2.id] #assign the created subnets to the acl network otherwirse the NACL is assigned to a default subnet

  tags = {
    Name = "production-network-acl"
  }
}

# Create acl rules for the network
# ACL inbound
resource "aws_network_acl_rule" "all_inbound_traffic_acl" {
  network_acl_id = aws_network_acl.production_acl_network.id
  rule_number    = 180
  protocol       = -1
  rule_action    = "allow"
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

# ACL outbound
resource "aws_network_acl_rule" "all_outbound_traffic_acl" {
  network_acl_id = aws_network_acl.production_acl_network.id
  egress         = true
  protocol       = -1
  rule_action    = "allow"
  rule_number    = 180
  cidr_block     = "0.0.0.0/0"
  from_port      = 0
  to_port        = 0
}

# ---------------------------------------- Step 3: Create security group/ rules ----------------------------------------
resource "aws_security_group" "sg_prod" {
    name   = "production-security-group"
    vpc_id = aws_vpc.vpc_prod.id
}

# Create first (inbound) security rule to open port 22 for ssh connection request
resource "aws_security_group_rule" "ssh_inbound_rule_prod" {
  type              = "ingress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"] #aws_vpc.vpc_prod.cidr_block, "0.0.0.0/0"
  security_group_id = aws_security_group.sg_prod.id
  description       = "security rule to open port 22 for ssh connection"
}

# Create fifth (inbound) security rule to allow pings of public ip address of ec2 instance from local machine
resource "aws_security_group_rule" "ping_public_ip_sg_rule" {
  type              = "ingress"
  from_port         = 8
  to_port           = 0
  protocol          = "icmp"
  cidr_blocks       = ["0.0.0.0/0"] #aws_vpc.vpc_prod.cidr_block, "0.0.0.0/0"
  security_group_id = aws_security_group.sg_prod.id
  description       = "allow pinging elastic public ipv4 address of ec2 instance from local machine"
}

#--------------------------------

# Create first (outbound) security rule to open port 80 for HTTP requests (this will help to download packages while connected to vm)
resource "aws_security_group_rule" "http_outbound_rule_prod" {
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"] #aws_vpc.vpc_prod.cidr_block, "0.0.0.0/0"
  security_group_id = aws_security_group.sg_prod.id
  description       = "security rule to open port 80 for outbound connection with http from remote server"
}

# Create second (outbound) security rule to open port 443 for HTTPS requests
resource "aws_security_group_rule" "https_outbound_rule_prod" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"] #aws_vpc.vpc_prod.cidr_block, "0.0.0.0/0"
  security_group_id = aws_security_group.sg_prod.id
  description       = "security rule to open port 443 for outbound connection with https from remote server"
}

# ---------------------------------------- Step 4: SSH key generated for accessing VM ----------------------------------------
resource "tls_private_key" "ssh_key_prod" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

# ---------------------------------------- Step 5: Generate aws_key_pair ----------------------------------------
resource "aws_key_pair" "generated_key_prod" {
  key_name   = "${var.prefix}-server-ssh-key"
  public_key = tls_private_key.ssh_key_prod.public_key_openssh

  tags   = {
    Name = "SSH key pair for production server"
  }
}

# ---------------------------------------- Step 6: Create network interface ----------------------------------------

# Create network interface
resource "aws_network_interface" "network_interface_prod" {
  subnet_id       = aws_subnet.subnet_prod.id
  security_groups = [aws_security_group.sg_prod.id]
  #private_ip      = aws_eip.prod_server_public_ip.private_ip #!!! not sure if this argument is correct !!!
  description     = "Production server network interface"

  tags   = {
    Name = "production-network-interface"
  }
}

# ---------------------------------------- Step 7: Create the Elastic Public IP after having created the network interface ----------------------------------------

resource "aws_eip" "prod_server_public_ip" {
  vpc               = true
  #instance          = aws_instance.production_server.id
  network_interface = aws_network_interface.network_interface_prod.id
  #don't specify both instance and a network_interface id, one of the two!

  depends_on        = [aws_internet_gateway.gw, aws_network_interface.network_interface_prod]
  tags   = {
    Name = "production-elastic-ip"
  }
}

# ---------------------------------------- Step 8: Associate public ip to network interface ----------------------------------------

resource "aws_eip_association" "eip_assoc" {
  #dont use instance, network_interface_id at the same time
  #instance_id   = aws_instance.production_server.id
  allocation_id = aws_eip.prod_server_public_ip.id
  network_interface_id = aws_network_interface.network_interface_prod.id

  depends_on = [aws_eip.prod_server_public_ip, aws_network_interface.network_interface_prod]
}

# ---------------------------------------- Step 9: Create route table with rules ----------------------------------------

resource "aws_route_table" "route_table_prod" {
  vpc_id = aws_vpc.vpc_prod.id
  tags   = {
    Name = "route-table-production-server"
  }
}

/*documentation =>
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html#Add_IGW_Routing
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html?icmpid=docs_ec2_console#ec2-instance-connect-setup-security-group
*/

resource "aws_route" "route_prod_all" {
  route_table_id         = aws_route_table.route_table_prod.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.gw.id
  depends_on             = [
    aws_route_table.route_table_prod, aws_internet_gateway.gw
  ]
}

# Create main route table association with the two subnets
resource "aws_main_route_table_association" "main-route-table" {
  vpc_id         = aws_vpc.vpc_prod.id
  route_table_id = aws_route_table.route_table_prod.id
}

resource "aws_route_table_association" "main-public-1-a" {
  subnet_id      = aws_subnet.subnet_prod.id
  route_table_id = aws_route_table.route_table_prod.id
}

resource "aws_route_table_association" "main-public-1-b" {
  subnet_id      = aws_subnet.subnet_prod_id2.id
  route_table_id = aws_route_table.route_table_prod.id
}

# ---------------------------------------- Step 10: Create the AWS EC2 instance ----------------------------------------

resource "aws_instance" "production_server" {
  depends_on                  = [aws_eip.prod_server_public_ip, aws_network_interface.network_interface_prod]
  ami                         = "ami-09e67e426f25ce0d7"
  instance_type               = "t2.micro"
  #key_name                    = "MyKeyPair"#aws_key_pair.generated_key_prod.key_name
  key_name                    = aws_key_pair.generated_key_prod.key_name

  network_interface {
    network_interface_id = aws_network_interface.network_interface_prod.id
    device_index         = 0
  }

  ebs_block_device {
    device_name = "/dev/sda1"
    volume_type = "standard"
    volume_size = 8
  }

  connection {
    type        = "ssh"
    host        = aws_eip.prod_server_public_ip.public_ip #Error: host for provisioner cannot be empty -> https://github.com/hashicorp/terraform-provider-aws/issues/10977
    user        = "ubuntu"
    private_key = tls_private_key.ssh_key_prod.private_key_pem
    timeout     = "1m"
  }

  provisioner "remote-exec" {
    inline = [
      "echo 'Installing modules...'",
      "sudo apt-get update",
      "sudo apt-get install -y openjdk-8-jdk",
      "sudo apt install -y python2.7 python-pip",
      "sudo apt install -y docker.io",
      "sudo systemctl start docker",
      "sudo systemctl enable docker",
      "pip install setuptools",
      "echo 'Modules installed via Terraform'"
    ]
    on_failure = fail
  }

  #user_data= <<-EOF
        #! /bin/bash
    #echo "Installing modules..."
    #sudo apt-get update
    #sudo apt-get install -y openjdk-8-jdk
    #sudo apt install -y python2.7 python-pip
    #sudo apt install -y docker.io
    #sudo systemctl start docker
    #sudo systemctl enable docker
    #pip install setuptools
    #echo "Modules installed via Terraform"
    #EOF

  tags   = {
    Name = "production-server"
  }

  volume_tags = {
    Name = "production-volume"
  }
}