vSphere Cloud-Init not keeping the static IP

Question

I'm getting odd behavior that I don't understand in dealing with vSphere and Terraform. My terraform code provisions a host in vSphere with a static network configuration (network, IP, mask, gateway, etc.), and on the first boot, it seems to be correct. The IP and relevant network settings are applied.

However on reboot, the network configuration falls back to DHCP, which fails.

I see in the /var/log/cloud-init.log file that on the first boot, it is able to successfully apply the config:

However after that, on reboot, it reverts to DHCP?

I noticed this issue when trying to bring up Consul on the host, and Consul complained that there wasn't an IPv4 address to bind to. So, when I rebooted, the IP was gone. So I don't think the address is applying correctly.

The terraform code works for DHCP based networks, but for some reason when I apply static configurations, it adds it once, and then doesn't seem like it keeps.

Anyone ever hit this similar issue?

Edit:

Troubleshooting further by reading the cloud-init debugs:

This is the first boot, which should be the base template.

The second boot, which should be the deployed host receiving the config from terraform -> cloud-init:

Answer 1

...it's a bug in vSphere.

https://kb.vmware.com/s/article/71264

 Symptoms
Virtual Machine has cloud-init customization enabled and a static IP
After reboot the virtual machine is configured with DHCP

 Cause
This issue occurs due to cloud-init considering the virtual machine as a new instance. This instance can not retrieve data from the datasource when there is no customization, and will have DatasourceNone applied defaulting to DHCP.

 Resolution
Currently there is no resolution.

 Workaround
To workaround this issue apply one of the following.workarounds:
Add the setting manual_cache_clean: True to the /etc/cloud/cloud.cfg for the customized virtual machine.
Uninstall cloud-init from the virtual machine