How to handle a broken instance after automatic migration

Question

This relates to debugging a VM in Google Compute Engine:

It seems a long running instance became inaccessible after a Google-based operation: "Automatically migrate an instance". Are there methods or means of debugging this? Rebooting the instance, trying to connect via SSH using in browser (or direct) means both fail. Status shows the instance as consuming CPU while loading then being mostly flat.

It seems as though the instance itself is permantently inaccessible. Has this come up for anyone else? Is there a simple status page to see if the instance is simply broken or if this is a service outage?

Automatic migration occurred 2014-08-25, and that likely was triggered by Google as part of planned (or unplanned) maintenance.

UPDATE: Following three hours of inability to connect, the networking issue was resolved. I've not found a status page, nor seen any mention of the outage.

Answer 1

Charles,

Have a look at your console output, either from the command line:

gcloud compute instances get-serial-port-output NAME [--project PROJECT_ID] [--zone ZONE]

(where NAME, PROJECT_ID, and ZONE match your VM) or from

https://console.developers.google.com/project/PROJECT_ID/compute/instancesDetail/zones/ZONE/instances/NAME

(scroll down to the bottom and click "View console").

That may give you an idea why it is not letting you access it. (I had configured an encrypted SSL server certificate in Apache2, and it wanted the pass phrase at boot time, but the VM's console is not interactive. It's read-only. I had to delete my VM but keep my boot disk, mount that boot disk on another VM, find the certificate file and convert it to unencrypted, then unmount that second disk and create a new VM using that now-fixed boot disk. Worked like a charm. Of course, your VM may complain about something different.)

-- Brian Haney
By day: Technical Solutions Engineer, Google Cloud Platform Support
(My opinions are MINE, not Google's.)
By night: WebOps consultant -- empowering people with Internet technology since 1993

Answer 2

Had the same problem, my instance was dead for 2 days (no SSH possible). Had to manually shutdown and restart.

Background: container VM image (for running docker); external hard disk attached

Serial console log:

Welcome to [1mContainer-Optimized OS from Google[0m!
[    8.457917] systemd[389]: [/usr/lib/systemd/system/[email protected]:6] Failed to add required mount for, ignoring: %I
[    8.461882] systemd[389]: [/usr/lib/systemd/system/[email protected]:6] Failed to add required mount for, ignoring: %I
[    8.465906] systemd[389]: [/usr/lib/systemd/system/[email protected]:6] Failed to add required mount for, ignoring: %I
[    8.471102] systemd[389]: usr-share-oem.mount: Unit is bound to inactive unit dev-sda8.device. Stopping, too.
[    8.472843] systemd[389]: home-chronos.mount: Unit is bound to inactive unit dev-sda1.device. Stopping, too.
[    8.474570] systemd[389]: home.mount: Unit is bound to inactive unit dev-sda1.device. Stopping, too.
[    8.476206] systemd[389]: mnt-stateful_partition.mount: Unit is bound to inactive unit dev-sda1.device. Stopping, too.
[    8.478911] systemd[389]: display-manager.service: Cannot add dependency job, ignoring: Unit display-manager.service failed to load: No such file or directory.
[    8.481490] systemd[389]: Started Dispatch Password Requests to Console Directory Watch.
[[32m  OK  [0m] Started Dispatch Password Requests to Console Directory Watch.
[    8.486205] systemd[389]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[[32m  OK  [0m] Set up automount Arbitrary Executab...ats File System Automount Point.
[    8.491072] systemd[389]: Started Forward Password Requests to Wall Directory Watch.
[[32m  OK  [0m] Started Forward Password Requests to Wall Directory Watch.
[    8.496038] systemd[389]: Reached target Paths.
[[32m  OK  [0m] Reached target Paths.
[    8.500025] systemd[389]: Reached target Swap.
[[32m  OK  [0m] Reached target Swap.
[    8.502994] systemd[389]: Reached target Remote File Systems.
[[32m  OK  [0m] Reached target Remote File Systems.
[    8.507099] systemd[389]: Created slice Root Slice.
[[32m  OK  [0m] Created slice Root Slice.
[    8.511319] systemd[389]: Created slice System Slice.
[[32m  OK  [0m] Created slice System Slice.
[    8.515294] systemd[389]: Created slice Slice for System Daemons.
[[32m  OK  [0m] Created slice Slice for System Daemons.
[    8.519358] systemd[389]: Created slice system-serial\x2dgetty.slice.
[[32m  OK  [0m] Created slice system-serial\x2dgetty.slice.
[    8.523943] systemd[389]: Created slice system-bindmount.slice.
[[32m  OK  [0m] Created slice system-bindmount.slice.
[    8.528063] systemd[389]: Listening on Journal Socket.
[[32m  OK  [0m] Listening on Journal Socket.
[    8.534049] systemd[389]: Starting Init GCI filesystems...
         Starting Init GCI filesystems...
[    8.538880] systemd[389]: Starting Create list of required static device nodes for the current kernel...
         Starting Create list of required st... nodes for the current kernel...
[    8.559447] systemd[389]: Mounting /mnt/disks...
         Mounting /mnt/disks...
[    8.604573] systemd[389]: Mounting /var/lib/cloud...
         Mounting /var/lib/cloud...
[    8.608462] systemd[389]: Mounting POSIX Message Queue File System...
         Mounting POSIX Message Queue File System...
[    8.613688] systemd[389]: Starting Apply Kernel Variables...
         Starting Apply Kernel Variables...
[    8.616680] systemd[389]: Listening on /dev/initctl Compatibility Named Pipe.
[[32m  OK  [0m] Listening on /dev/initctl Compatibility Named Pipe.
[    8.620556] systemd[389]: Listening on Journal Socket (/dev/log).
[    8.622630] Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.
[[32m  OK  [0m] Listening on Journal Socket (/dev/log).
[    8.627801] systemd[389]: Listening on Journal Audit Socket.
[[32m  OK  [0m] Listening on Journal Audit Socket.
[    8.633048] systemd[389]: Listening on udev Control Socket.
[[32m  OK  [0m] Listening on udev Control Socket.
[    8.637607] systemd[389]: Starting Resize stateful partition...
         Starting Resize stateful partition...
[    8.642185] systemd[389]: Starting Remount Root and Kernel File Systems...
         Starting Remount Root and Kernel File Systems...
[    8.645313] systemd[389]: Starting Journal Service...
         Starting Journal Service...
[    8.648043] systemd[389]: Created slice User and Session Slice.
[[32m  OK  [0m] Created slice User and Session Slice.
[    8.652056] systemd[389]: Reached target Slices.
[[32m  OK  [0m] Reached target Slices.
[    8.659068] systemd[389]: Listening on networkd rtnetlink socket.
[[32m  OK  [0m] Listening on networkd rtnetlink socket.
[    8.663053] systemd[389]: Listening on udev Kernel Socket.
[[32m  OK  [0m] Listening on udev Kernel Socket.
[    8.668255] systemd[389]: Starting udev Coldplug all Devices...
         Starting udev Coldplug all Devices...
[    8.681967] systemd[389]: Mounted /var/lib/cloud.
[[32m  OK  [0m] Mounted /var/lib/cloud.
[    8.685042] systemd[389]: Mounted POSIX Message Queue File System.
[[32m  OK  [0m] Mounted POSIX Message Queue File System.
[    8.690680] systemd[389]: Mounted /mnt/disks.
[[32m  OK  [0m] Mounted /mnt/disks.
[    8.701337] systemd[389]: Started Init GCI filesystems.
[[32m  OK  [0m] Started Init GCI filesystems.
[    8.706368] systemd[389]: Started Create list of required static device nodes for the current kernel.
[[32m  OK  [0m] Started Create list of required sta...ce nodes for the current kernel.
[    8.712553] systemd-journald[430]: File /var/log/journal/d4fa6498b846d6212ac87e865788a99d/system.journal corrupted or uncleanly shut down, renaming and replacing.
[    8.715515] systemd[389]: Started Apply Kernel Variables.
[[32m  OK  [0m] Started Apply Kernel Variables.
[    8.719634] systemd[389]: Started Remount Root and Kernel File Systems.
[[32m  OK  [0m] Started Remount Root and Kernel File Systems.
[    8.764898] systemd[389]: Started udev Coldplug all Devices.
[[32m  OK  [0m] Started udev Coldplug all Devices.
[    8.788181] systemd[389]: Starting Rebuild Dynamic Linker Cache...
         Starting Rebuild Dynamic Linker Cache...
[    8.791452] systemd[389]: Starting Create Static Device Nodes in /dev...
         Starting Create Static Device Nodes in /dev...
[    8.834830] systemd[389]: Started Journal Service.
[[32m  OK  [0m] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[    8.901885] systemd-journald[430]: Received request to flush runtime journal from PID 389
[[32m  OK  [0m] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[[32m  OK  [0m] Reached target Local File Systems (Pre).
         Starting [email protected]...
         Starting [email protected]...
         Starting [email protected]...
[[32m  OK  [0m] Started [email protected].
[[32m  OK  [0m] Started [email protected].
[[32m  OK  [0m] Started [email protected].
         Starting Mount /var/lib/docker with 'exec'...
[    9.032344] EXT4-fs (sda1): re-mounted. Opts: commit=30,data=ordered
         Starting Mount /var/lib/google with 'exec'...
[    9.036810] EXT4-fs (sda1): re-mounted. Opts: commit=30,data=ordered
[    9.040218] EXT4-fs (sda1): re-mounted. Opts: commit=30,data=ordered
         Starting Mount /var/lib/toolbox with 'exec' and 'suid' bits...
[[32m  OK  [0m] Started Mount /var/lib/docker with 'exec'.
[[32m  OK  [0m] Started Mount /var/lib/google with 'exec'.
[[32m  OK  [0m][    9.089836] systemd-udevd[456]: IMPORT{builtin}: 'blkid --offset=$env{ID_CDROM_MEDIA_SESSION_LAST_OFFSET}' unknown /lib/udev/rules.d/60-persistent-storage.rules:60
 Started Mount /var/lib/toolbox with 'exec' and 'suid' bits.
[    9.101830] systemd-udevd[456]: IMPORT{builtin}: 'blkid --noraid' unknown /lib/udev/rules.d/60-persistent-storage.rules:63
[    9.113855] systemd-udevd[456]: IMPORT{builtin}: 'blkid' unknown /lib/udev/rules.d/60-persistent-storage.rules:66
[[32m  OK  [0m] Started Flush Journal to Persistent Storage.
[[32m  OK  [0m] Started udev Kernel Device Manager.
[    9.252609] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[[32m  OK  [0m] Found device /dev/ttyS0.
[[32m  OK  [0m] Reached target Local File Systems.
         Starting Rebuild Journal Catalog...
         Starting Create Volatile Files and Directories...
[    9.340589] AVX version of gcm_enc/dec engaged.
[    9.341526] AES CTR mode by8 optimization enabled
[[32m  OK  [0m] Started Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Load Security Auditing Rules...
[[32m  OK  [0m] Started Resize stateful partition.
[[32m  OK  [0m] Started Rebuild Journal Catalog.
[[32m  OK  [0m] Started Network Time Synchronization.
[[32m  OK  [0m] Reached target System Time Synchronized.
[[32m  OK  [0m] Started Load Security Auditing Rules.
[[32m  OK  [0m] Started Rebuild Dynamic Linker Cache.
         Starting Update is Completed...
[[32m  OK  [0m] Started Update is Completed.
[[32m  OK  [0m] Reached target System Initialization.
[[32m  OK  [0m] Started Run Crash Sender hourly.
[[32m  OK  [0m] Listening on D-Bus System Message Bus Socket.
         Starting Docker Socket for the API.
[[32m  OK  [0m] Started Daily Cleanup of Temporary Directories.
[[32m  OK  [0m] Reached target Timers.
[[32m  OK  [0m] Listening on Docker Socket for the API.
[[32m  OK  [0m] Reached target Sockets.
[[32m  OK  [0m] Reached target Basic System.
         Starting Configure ip6tables...
         Starting Permit User Sessions...
         Starting Initialize Crash Reporter...
         Starting Configure iptables...
         Starting Initial cloud-init job (pre-networking)...
         Starting Login Service...
[[32m  OK  [0m] Started D-Bus System Message Bus.
[   10.185174] ip6_tables: (C) 2000-2006 Netfilter Core Team
         Starting Network Service...
         Starting Notify Upstart about D-Bus service...
[[32m  OK  [0m] Started Configure ip6tables.
[[32m  OK  [0m] Started Permit User Sessions.
[[32m  OK  [0m] Started Notify Upstart about D-Bus service.
[   10.297969] init: failsafe-delay main process (669) killed by TERM signal
[[32m  OK  [0m] Started Network Service.
[[32m  OK  [0m] Started Login Service.
         Starting Wait for Network to be Configured...
[[32m  OK  [0m] Started Serial Getty on ttyS0.
[[32m  OK  [0m] Reached target Login Prompts.
[[32m  OK  [0m] Started Wait for Network to be Configured.
         Starting Hostname Service...
[[32m  OK  [0m] Started Hostname Service.
[[32m  OK  [0m] Started Initialize Crash Reporter.
         Starting Run per-boot crash collection tasks...
[[32m  OK  [0m] Started Run per-boot crash collection tasks.
[[32m  OK  [0m] Started Configure iptables.
[[32m  OK  [0m] Reached target Network.
[[32m  OK  [0m] Reached target Network is Online.
         Starting Initialize device policy...
         Starting Docker Setup for GKE...
[[32m  OK  [0m] Started Google Compute Engine User Accounts Manager Daemon.
         Starting Network Name Resolution...
         Starting Google Compute Engine VM initialization...
[[32m  OK  [0m] Started Google Compute Engine Address Manager Daemon.
         Starting OpenSSH server daemon...
[[32m  OK  [0m] Started Network Name Resolution.
[[32m  OK  [0m] Started OpenSSH server daemon.
[[32m  OK  [0m] Started Docker Setup for GKE.
         Starting Docker Application Container Engine...