How to save data from a crashed Cloud SQL instance without data loss

0
votes

My Cloud SQL instance(MySQL) with High Availability enabled is totally down now - I can't connect into MySQL service, mysqldump, Export and Clone on the console are all down, even the Stop or Restart button response nothing but error messages.

Is there any solution to save my data without data loss? If I restore instance from the last backup, it will result a 12 hours data loss, disaster.

I considered "point-in-time recovery", but the question is how to locate the binary log filename and recover position if I can't enter into MySQL.

the error log from the crashed instance:

2021-01-26 19:03:29.613 CST
2021-01-26T11:03:29.612303Z 0 [ERROR] InnoDB: Invalid redo log header checksum.
Error
2021-01-26 19:03:29.613 CST
2021-01-26T11:03:29.612315Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
Error
2021-01-26 19:03:30.213 CST
2021-01-26T11:03:30.213169Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
Error
2021-01-26 19:03:30.214 CST
2021-01-26T11:03:30.214078Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Error
2021-01-26 19:03:30.214 CST
2021-01-26T11:03:30.214456Z 0 [ERROR] Failed to initialize builtin plugins.
1
mysqlgoogle-cloud-platforminnodbgoogle-cloud-sql
The instance just recovered - after 4 hours downtime. By the time I was trying to apply the point-in-time recovery in a cloned instance, result unexpectedly the point-in-time job failed, but the original instance mysteriously recovered itself. I will quit Cloud SQL and move my DB into my GKE cluster tomorrow. And, ask for refunds. – Mou Hsiao
Do you have a replica? If so, you can trry to promote it. – guillaume blaquiere
Thank you, I just lunched one, though it's weird cause I think HA failover should be responsible for it, not a read replica. I will try moving to legacy HA later. – Mou Hsiao

1 Answers

1
votes

Instances configured with High Availability should fail over to the secondary instance in a matter of minutes. While this operation is taking place is not possible to perform exports or other operations.

If the failover took 4 hour you may be eligible for a credit adjustment according to the Cloud SQL SLA. I would suggest to open support ticket if you would like to explore such path.

Lastly, I would not advise to use the legacy HA configuration as it won't be available anymore past Q1 2021 as mentioned here.