Can connect with existing SSH key but not a new one

1
votes

I'm trying to give access to a GCE VM by adding SSH Keys to the project metadata. My current SSH key is in the project metadata and I can connect just fine using:

ssh -i ~/.ssh/<private_key> <username>@<instance_ip>

Now, I generated another key:

ssh-keygen -t rsa -f ~/.ssh/<new_key> -C <new_username>

After adding the generated public key to the project metadata, I then run:

ssh -i ~/.ssh/<new_private_key> <new_username>@<instance_ip>

But I get Permission denied (publickey,gssapi-keyex,gssapi-with-mic). Running with -vvv flag doesn't show me much besides the key being rejected.

Things I know/checked:

  • firewall isn't an issue because I can connect using my original key from same location
  • the instance is running SSH (running nc <instance_nat_ip> 22 shows "OpenSSH" etc.)
  • no passphrases were used with the generation of any SSH key
  • there are no instance-level restrictions on project-wide metadata
  • there are no instance-level ssh keys already added
  • there are no newlines/breaks causing the key to be malformed
  • permissions on ~./ssh aren't an issue since another key pair works fine from the same directory, additionally, both key pairs have the same permissions anyways
  • OSLogin isn't enabled either on the project or instance

Things I've tried:

  • removing and readding the SSH keys in project metadata
  • trying with new key pairs generate on another person's machine
  • restarting sshd service

Questions:

  • Does the username specified during the ssh-keygen step have to already exist on the remote instance prior to adding the key to the metadata? i.e. do I have to run sudo useradd <new_username> while SSH'd into the instance Creating a new test instance revealed this to not be the case, all users in the project metadata were created automatically
  • why does my existing SSH key work and not new ones even though they are added the same way?
  • there's a chance the enable-oslogin:TRUE was applied to the instance briefly a long time ago (I'm not sure since I'm not the one who created the instance) but it's no longer there in the instance or project metadata. Would having that been enabled, even briefly, cause some issues?

EDIT: I started up a new instance in the same project with the same network details and I was able to SSH to that instance using the new key. Original instance is still denying the key

1
google-cloud-platformgoogle-compute-engine
Look at the sshd logfile. - John Hanley
Did you do any updates on the instance? - night-gold

1 Answers

2
votes

Did some digging around and found out that the systemd service that propagates accounts information from the metadata server is a daemon called google-accounts-daemon.

When I ran sudo ps aux | grep daemon I didn't see it running as I did on the test instance I created.

So when I ran sudo systemctl restart google-accounts-daemon the SSH keys magically propagated and everything worked.

I have no idea what caused the daemon to stop running in the first place, so if anyone has ideas, that'd be appreciated in case this comes up in the future.