SSH using terminal, cloud shell, or window browser in to Google Cloud VM instance not working

Question

I am new to google cloud computing. Recently have not been able to SSH into my VM instance, previously it worked by clicking the SSH button and opened a new browser window. I have tried several fixes based on internet searches, mostly summarized below:

When I click the SSH button, it says "Transferring Keys to the VM" but never logs in and eventually times out.

Attempt 1: I attempted to follow another thread Google VM Instance not opening with SSH and I increased the persistent disk size and reset the VM. Now it says "Could not connect, retrying..." when I click the SSH button and eventually times out.

Attempt 2: I also tried SSHing into the VM using Google Cloud Shell using the command

gcloud beta compute ssh --zone "myZONE" "VM_instance_name" --project "PROJECT_ID"

And got the following error

Permission denied (publickey).
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].

I created new keys using

sudo gcloud compute config-ssh

and it gave me this error

ERROR: (gcloud.compute.config-ssh) The project property is set to the empty string, which is invalid.

I ran the following and it seems to work

gcloud config set project myPROJECT_ID
Updated property [core/project].

But when I ran sudo gcloud again it gave me the same empty string error.

Attempt 3: I also tried setting up the log in locally using google-cloud-sdk. I followed interactive installing instructions on Using the Google Cloud SDK installer. I tried obtaining ssh key using

gcloud compute project-info describe --project myPROJECT_ID

and copied in into the ssh key on for the VM thru the cloud console website. I also tried

sudo gcloud compute config-ssh

which seemed to work and gave the following

Updating project ssh metadata...⠼Updated [https://...].                                                 
Updating project ssh metadata...done.                                                                                                             
You should now be able to use ssh/scp with your instances.
For example, try running:

  $ ssh VMinstance.myzone.PROJECT_ID

When I try running the ssh command I get the following error.

ssh: Could not resolve hostname VMinstance.myzone.PROJECT_ID: nodename nor servname provided, or not known

My instance has "enable connecting to serial ports" activated.

Any assistance is greatly appreciated. Thank you in advance.

Update: for the VM instance in question, I clicked on serial port 1 (console) and here are the first 100 lines

serialport: Connected to PROJECT_ID.Zone.VMInstance port 1 (session ID: ##..., active connections: 1).
Jul  2 12:19:02 INSTANCE google-accounts: INFO Removing user root.
Jul  2 12:19:02 INSTANCE google-accounts: INFO Removing user root from the Google sudoers group.
[1##.##8] google_accounts_daemon[822]: Removing user root from group google-sudoers
Jul  2 12:19:02 INSTANCE google_accounts_daemon[822]: Removing user root from group google-sudoers
[1##.##0] google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device
Jul  2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device
[1##.##6] google_accounts_daemon[822]: gpasswd: cannot lock /etc/group; try again later.
Jul  2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: cannot lock /etc/group; try again later.
Jul  2 12:19:02 INSTANCE google-accounts: WARNING Could not update user root. Command '['gpasswd', '-d', 'root', 'google-sudoers']' returned non-zero exit status 1..
Jul  2 12:19:02 INSTANCE google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_watcher.py", line 200, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 285, in HandleAccounts#012    self.utils.SetConfiguredUsers(desired_users.keys())#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_utils.py", line 318, in SetConfiguredUsers#012    mode='w', prefix=prefix, delete=True) as updated_users:#012  File "/usr/lib/python3.6/tempfile.py", line 681, in NamedTemporaryFile#012    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)#012  File "/usr/lib/python3.6/tempfile.py", line 269, in _sanitize_params#012    dir = gettempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 437, in gettempdir#012    tempdir = _get_default_tempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 372, in _get_default_tempdir#012    dirlist)#012FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
Jul  2 12:19:41 INSTANCE systemd[1]: snapd.service: Start operation timed out. Terminating.
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/ssh/sedYpckqQ: No space left on device
[1##.##1] google_accounts_daemon[822]: sed: couldn't flush /etc/ssh/sedYpckqQ: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedeXXx0O: No space left on device
[1##.##4] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedeXXx0O: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed27Z7HO: No space left on device
[1##.##2] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed27Z7HO: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed3bBBFO: No space left on device
[1##.##6] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed3bBBFO: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedUKduxP: No space left on device
[1##.##3] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedUKduxP: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedhtixlP: No space left on device
[1##.##0] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedhtixlP: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedy6zVAS: No space left on device
[1##.##9] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedy6zVAS: No space left on device
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: Restarting optional services.
[1##.##7] google_accounts_daemon[822]: Restarting optional services.
Jul  2 12:20:03 INSTANCE systemd[1]: Stopping Login Service...
Jul  2 12:20:03 INSTANCE systemd[1]: Stopped Login Service.
Jul  2 12:20:03 INSTANCE systemd[1]: Starting Login Service...
Jul  2 12:20:03 INSTANCE systemd[1]: Started Login Service.
Jul  2 12:20:03 INSTANCE systemd[1]: Stopping Regular background program processing daemon...
Jul  2 12:20:03 INSTANCE systemd[1]: Stopped Regular background program processing daemon.
Jul  2 12:20:03 INSTANCE systemd[1]: Started Regular background program processing daemon.
Jul  2 12:20:03 INSTANCE google_accounts_daemon[822]: Restarting SSHD
[1##.##5] google_accounts_daemon[822]: Restarting SSHD
Jul  2 12:20:03 INSTANCE systemd[1]: Stopping OpenBSD Secure Shell server...
Jul  2 12:20:03 INSTANCE systemd[1]: Stopped OpenBSD Secure Shell server.
Jul  2 12:20:03 INSTANCE systemd[1]: Starting Google Compute Engine Instance Setup...
Jul  2 12:20:03 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul  2 12:20:03 INSTANCE instance-setup: INFO Running google_set_multiqueue.
Jul  2 12:20:03 INSTANCE instance-setup: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Jul  2 12:20:03 INSTANCE instance-setup: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Jul  2 12:20:03 INSTANCE instance-setup: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Jul  2 12:20:03 INSTANCE instance-setup: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Jul  2 12:20:03 INSTANCE instance-setup: INFO /usr/bin/google_set_multiqueue: line 139: echo: write error: No such file
 or directory
Jul  2 12:20:03 INSTANCE instance-setup: INFO cat: /sys/class/net/ens4/queues/tx-0/xps_cpus: No such file or directory
Jul  2 12:20:03 INSTANCE instance-setup: INFO Queue 0 XPS=/sys/class/net/ens4/queues/tx-0/xps_cpus for
Jul  2 12:20:03 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul  2 12:20:03 INSTANCE systemd[1]: Started Google Compute Engine Instance Setup.
Jul  2 12:20:03 INSTANCE systemd[1]: Starting OpenBSD Secure Shell server...
Jul  2 12:20:03 INSTANCE systemd[1]: Started OpenBSD Secure Shell server.
Jul  2 12:20:03 INSTANCE systemd[1]: Stopping OpenBSD Secure Shell server...
Jul  2 12:20:03 INSTANCE systemd[1]: Stopped OpenBSD Secure Shell server.
Jul  2 12:20:03 INSTANCE systemd[1]: Starting Google Compute Engine Instance Setup...
Jul  2 12:20:04 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul  2 12:20:04 INSTANCE instance-setup: INFO Running google_set_multiqueue.
Jul  2 12:20:04 INSTANCE instance-setup: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Jul  2 12:20:04 INSTANCE instance-setup: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Jul  2 12:20:04 INSTANCE instance-setup: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Jul  2 12:20:04 INSTANCE instance-setup: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Jul  2 12:20:04 INSTANCE instance-setup: INFO /usr/bin/google_set_multiqueue: line 139: echo: write error: No such file or directory

Answer 1

If your instance is running and you can restart it you may try logging in via serial console. This method is independed from any firewall / network settings :)

Just add a user with a password; create a startup script like this (it will create a user and add it to google-sudoers group so you can do everything with this account):

#! /bin/bash
adduser username
echo 'sudouser:userspass' | chpasswd
usermod -aG google-sudoers sudouser

And then connect to serial console you can log in. Either using console window or cloud shell: gcloud compute connect-to-serial-port instance-name

This will allow you to log in. Unless there's something really wront with your VM.

But - judging from you desription - you want regular SSH access.

For that make sure that:

• you firewall rules don't block port 22 on this machine
• your firewall on a VM in question allows traffic on port 22
• your ethernet interface is up in the VM
• SSH server listens (ps aux | grep sshd) on port 22

Then check if your VM has external IP - if not you will be only able to connect to your VM via another that has one, or via cloud shell (and of course serial console as described above).

However - since you got this Permission denied (publickey) message that means that for some reason your SSH keys don't work (corrupted, deleted etc). When you log in with serial console check if they're there. If not you can add them manually.

----------- UPDATE ------------

Look at this line: Jul 2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device - it says that your VM ran out of disk space. Stop your VM, try resizing your VM's disk first with gcloud compute disks resize mydiskname --size=100GB --zone=us-central1-a and start it. After the start VM's partition will be resized. –