Compute Engine HTTP Load Balancer, Instance Not Being Checked

0
votes

I have http load balancing working without issue on GCE however I've been having issues with the health-checks recently - my instances aren't being checked.

After updating an instance group with a new image (for example), I generally restore my instances with a reboot since I have a start-up script that runs chef.

I can see chef runs fine on the instances and I'm able to access the instances on their ephemeral ips remotely. My other instance groups are all fine.

The problem is that the health checks never run on these instances and they're never added back. I've set the cool down period to 480 seconds per group since chef takes up to 8 mins sometimes to run and complete.

The only way I can get the instances back is to edit the instance group - this seems to force it through.

Or, sometimes if I wait 15 mins, the instance finally gets checked.

Have I missed something here? Is there a something I need to do to officially get the instances back quicker.

2
httpload-balancinggoogle-compute-engine

2 Answers

0
votes

I also have this issue since a few weeks. If I recreate an instance from my instance group it will take up to 20 minutes before the load balancer detects that it's healthy. I have created a support ticket at Google and the support people where able to reproduce the issue. I've got earlier this week an update that they fixed it in the code and that I get another update if they pushed the fix to live. I guess this will be soon.

0
votes

@simonmorley: could you please describe your issues (here or in separate thread) with scaling down the instances? Now I do not know enough to be able to help. Just to clarify:

  • the "cooldown" parameter you mentioned is an Autoscaler's setting and is not related to Health Checks (it is just a hint for Autoscaler to ignore VM's data for this period as VM is booting up)
  • unfortunately currently instance restart does not reset the cooldown
  • Health Check seems to not catch up with instances and that sounds worrying. Did your restart involve changes in network configuration? How are you performing the restart?
  • do by "add back" you mean adding back to the load balanced pool?

Note: I'm on the Autoscaler team.