Deploy CoreOS virtual machine on vSphere with Terraform

Question

I'm having a really difficult time trying to deploy a CoreOS virtual machine on vsphere using Terraform.

So far this is the terraform file I'm using:

# Configure the VMware vSphere Provider. ENV Variables set for Username and Passwd.

provider "vsphere" {
 vsphere_server = "192.168.105.10"
 allow_unverified_ssl = true
}

provider "ignition" {
  version = "1.0.0"
}

data "vsphere_datacenter" "dc" {
  name = "Datacenter"
}

data "vsphere_datastore" "datastore" {
  name          = "vol_af01_idvms"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = "Cluster_rnd/Resources"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_network" "network" {
  name          = "VM Network"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_virtual_machine" "template" {
  name          = "coreos_production"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

# Create a folder
resource "vsphere_folder" "TestPath" {
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
  path       = "Test"
  type       = "vm"
}

#Define ignition data
data "ignition_networkd_unit" "vmnetwork" {
  name  = "00-ens192.network"

  content = <<EOF
  [Match]
  Name=ens192
  [Network]
  DNS=8.8.8.8
  Address=192.168.105.27/24
  Gateway=192.168.105.1
EOF
}

data "ignition_config" "node" {
  networkd = [
    "${data.ignition_networkd_unit.vmnetwork.id}"
  ]
}

# Define the VM resource
resource "vsphere_virtual_machine" "vm" {
 name   = "terraform-test"
 folder = "${vsphere_folder.TestPath.path}"
 resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
 datastore_id     = "${data.vsphere_datastore.datastore.id}"

 num_cpus = 2
 memory   = 1024
 guest_id = "other26xLinux64Guest"

 network_interface {
   network_id = "${data.vsphere_network.network.id}"
 }

 disk {
    name             = "terraform-test.vmdk"
    size             = "${data.vsphere_virtual_machine.template.disks.0.size}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}"
  }

 clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
  }

 extra_config {
    guestinfo.coreos.config.data.encoding = "base64"
    guestinfo.coreos.config.data          = "${base64encode(data.ignition_config.node.rendered)}"
  }
}

I'm using terraform vsphere provier to create the virtual machine and ignition provider to pass customization details of the virtual machine such as network configuration.

It is not quite clear to me if I'm using correctly the extra_config property on the virtual machine definition. You can find documentation about that property here.

Virtual machine gets created, but network settings never are applied, meaning that ignition provisioning is not correctly working.

I would appreciate any guidance on how to properly configure Terraform for this particular scenario (Vsphere environment and CoreOS virtual machine), specially regarding guestinfo configuration.

• Terraform v0.11.1, provider.ignition v1.0.0, provider.vsphere v1.1.0

• VMware ESXi, 6.5.0, 5310538

• CoreOS 1520.0.0

Answer 1

EDIT (2018-03-02)

As of version 1.3.0 of terraform vsphere provider, there is available a new vApp property. Using this property, there is no need to tweak the virtual machine using VMware PowerCLI as I did in the first answer.

There is a complete example of using this property here

The machine definition now would look something like this:

  ...

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
  }

  vapp {
     properties {
        "guestinfo.coreos.config.data.encoding" = "base64"
        "guestinfo.coreos.config.data"          = "${base64encode(data.ignition_config.node.rendered)}"
  }

  ...

OLD ANSWER

Finally got this working.

The workflow I have used to create a CoreOS machine on vSphere using Terraform is as follows:

1. Download the latest Container Linux Stable OVA from https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ova.
2. Import coreos_production_vmware_ova.ova into vCenter.
3. Edit machine settings as desired (number of CPUs, disk size etc.)
4. Disable "vApp Options" of virtual machine.
5. Convert the virtual machine into a virtual machine template.

Once you have done this, you've got a CoreOS virtual machine template that is ready to be used with Terraform.

As I said in a comment to the question, some days ago I found this and that lead me to understand that my problem could be related to not being able to perform step 4.

The thing is that to be able to disable "vApp Options" (i.e. to see in the UI the "vApp Options" tab of the virtual machine) you need DSR enabled in your vSphere cluster, and, to be able to enable DSR, your hosts must be licensed with a key that supports DRS. Mine weren't, so I was stuck in that 4th step.

I wrote to VMware support, and they told me an alternate way to do this, without having to buy a different license.

This can be done using VMware PowerCLI. Here are the steps to install PowerCLI, and here is the reference. Once you got PowerCLI installed, this is the script I used to disable "vApp Options" in my machines:

Import-Module VMware.PowerCLI

#connect to vcenter
Connect-VIServer -Server yourvCenter -User yourUser -Password yourPassword

#Use this to disable the vApp functionality.
$disablespec = New-Object VMware.Vim.VirtualMachineConfigSpec
$disablespec.vAppConfigRemoved = $True

#Use this to enable
$enablespec = New-Object VMware.Vim.VirtualMachineConfigSpec
$enablespec.vAppConfig = New-Object VMware.Vim.VmConfigSpec

#Get the VM you want to work against.
$VM = Get-VM yourTemplate | Get-View

#Disables vApp Options
$VM.ReconfigVM($disablespec)

#Enables vApp Options
$VM.ReconfigVM($enablespec)

I executed that on a Powershell and managed to reconfigure the virtual machine, performing that 4th step. With this, I finally got my CoreOS virtual machine template correctly configured for this scenario.

I've tested this with terraform vSphere provider versions v0.4.2 and v1.1.0 (the syntax changes) and the machine gets created correctly; Ignition provisioning works and everything you put on your ignition file (network configs, users etc.) is applied on the newly created machine.