To understand how Google Compute Engine manages the ssh keys, you have to understand how GCE manages the metadata (since, as you wrote, they are in the metadata store).
And more specifically, the difference between project and instance metadata is crucial. To quote the documentation (see previous links):
Metadata can be assigned at both the project and instance level. Project level metadata propagates to all virtual machine instances within the project, while instance level metadata only impacts that instance. You can set both project and instance level metadata but if you set the same key for both your project and instance metadata, Compute Engine will use the instance metadata.
While this seems rather logical and straightforward, one has to pay attention, very closely, to the used terms:
Project level metadata propagates to all virtual machine instances within the project [...]
and
You can set both [...] but if you set the same key for both [...], Compute Engine will use the instance metadata.
If you consider both assertions, it means two things:
- If you set the metadata at the project level ONLY, it will propagate in your instances.
- If you set the metadata at the instance level, it will take precedence over the project level ones, and nothing will be propagated.
As a direct consequence of that, the GCE platform takes care of placing/removing your ssh keys in the instance (and creating the relevant users when placing them, while just removing the key from the ~user/.ssh/authorized_keys file when removing them - so you don't lose any data for ~user) ONLY when you do not specify your own keys (at instance creation or later). If you do, the GCE platform will consider the ssh key management as manual, and nothing will be kept in sync with the metadata store.
Fortunately, the GCE platform is well done, and therefore, you need not re-creating an instance to get your keys managed by the GCE platform: you only need to remove the instance level metadata relative to the sshKeys.
The same way, if you add some instance level metadata with the key sshKeys, it will disable the ssh keys GCE platform management; unless you remove that instance level metadata.
Concerning the delay question: I didn't have any delay other than the network delay (so no platform execution delay noticeable) so far. I don't think it's impossible that the platform has delays from time to time, but it doesn't seem very likely to be the cause of your problem.
Additional note:
Some distributions (such as ubuntu) include a specific user (in the case of ubuntu: ~ubuntu), with which every user existing in the project-level ssh keys can login; but that user's authorized_keys is generated at instance creation time, and never seems to be changed again by the GCE platform. IMHO, the automatic ssh key management should be preferred.
Source: personal experience with GCE, terraform, and the Google Developer Console
sshKeysmetadata value for project-wide metadata has been deprecated; use thessh-keysmetadata value instead. More on this here. – Mike