Is it possible to snapshot a Google Cloud Spanner Database/table(s)? For compliance reasons we have to have daily snapshots of the current database that can be rolled back to in the event of a disaster: is this possible in Spanner? Is there intention to support it if not?
For those who might ask why we would need it as Spanner is replicated/redundant etc - it doesn't guard against human error (dropping a table by accident) or sabotage/espionage hence the question and requirement.
Thanks, M
Today, you can stream out a consistent snapshot by reading out all the data using your favorite tool (mapreduce, spark, dataflow) and reads at a specific timestamp (using Timestamp Bounds).
https://cloud.google.com/spanner/docs/timestamp-bounds
You have about an hour to do the export before the data gets garbage collected.
In the future, we will provide a Apache Beam/Dataflow connector to do this in a more scalable fashion. This will be our preferred method for doing import/export of data into Cloud Spanner.
Longer term, we will support backups and the ability to restore to a backup but that functionality is not currently available.
As of July 2018, Cloud Spanner now offers Import and Export functionality which allows you to export a database into Avro format. If you go to the specific Cloud Spanner database in question via the Google Cloud Console website, you will see Import and Export buttons toward the top. Simply click Export, populate the requested information such as a destination Google Cloud Storage bucket, and the database will be backed-up in Avro format to Google Cloud Storage. If you need to restore a database, use the corresponding Import functionality from the Cloud Spanner section of the Google Cloud Console website.
Note: The actual backup and restore (i.e., export and import) are done using Google Cloud Dataflow and you will be charged for the dataflow operation.
Google Cloud Spanner now has two methods that can be used to do backups.
https://cloud.google.com/spanner/docs/backup
You can either use the built-in backups or do an export/import using a dataflow job.
Per the thread below (also answered by eb80), I have been able to successfully script my backups from Spanner using their Import/Export DataFlow jobs by building the templates and running the gcloud command out of a cron job. The catch is that when you schedule the job you need to parse the output and get the job ID so that you can then check the status of it (again, I scripted that) and when you get JOB_STATE_DONE or JOB_STATE_FAILED, it's complete. If you need to run the Import, you'll need that job ID because the specific backup folder structure is:
gs://bucketName/yourFolder/instanceId-databaseId-jobId
How to batch load custom Avro data generated from another source?
