Azure storage account backup (tables and blobs)

Question

I need to periodically backup all blobs and tables in an Azure storage account so that we can restore all that data at a later time if we for any reason corrupt our data.

While I trust that data that we store in Azure is durable and recoverable in case of data center failures, we still need data in our storage accounts to be backed up to prevent from accidental overwrites and deletions (the human error factor).

We have implemented a solution for this that periodically lists all blobs and copies them over to a backup storage account. When a blob has been modified or deleted we simply create a snapshot of the old version in the backup account.

This approach has worked OK for us. But it only handles blobs, not table entities. We now need to support backing up table entities too.

Faced with this task now, I'm thinking that someone else probably have had this requirement before and come up with a smart solution. Or maybe there are commercial products that will do this?

It is not a requirement that the backup target is another Azure storage account. All we need is a way to recover all blobs and tables as they were at the time we ran the backup.

Any help is appreciated!

Answer 1

There are a variety of ways this can be handled.

If you want to do this on your own you can use the storage libraries and write code to just run through the table and pull down the data.

There are also a few services that can do this for you as well (FULL Disclosure: I work for a company that provides this as a service). Here is an article by Troy Hunt talking about our option: http://www.troyhunt.com/2014/01/azure-will-save-you-from-unexpected_28.html. We also have PowerShell Cmdlets that can pull table data down for you (cerebrata.com). To be fair we are not the only players in this space and there are others who have similar services.

Finally, at Tech Ed they announced that the AZCopy tool will be updated later this year so that it can pull down entire tables, which is just automating the reading through tables and pulling them down. There is currently no way to "Snapshot" a table so all of the methods above will result in a copy as the data is copied over, it might have changed in the source table by the time the copy is completed.

Answer 2

I've recently put together a simple solution to backup table storage. It uses the AzCopy tool and the Storage Rest Api to pull down a list of all the tables and do a backup to JSON.

Hope it's useful!

param(
[parameter(Mandatory=$true)]
[string]$Account,
[parameter(Mandatory=$true)]
[string]$SASToken,
[parameter(Mandatory=$true)]
[string]$OutputDir
)
$ErrorActionPreference = "Stop"

##Example Usage
#.\Backup-TableStorage.ps1 -OutputDir "d:\tablebackup" -Account "examplestorageaccount" -SASToken "?sv=2015-04-05&ss=t&srt=sco&sp=rl&st=2016-04-08T07%3A44%3A00Z&se=2016-04-09T07%3A55%3A00Z&sig=CNotAREALSIGNITUREBUTYOURESWOUDLGOHERE3D"

if (-not (Test-Path "${env:ProgramFiles(x86)}\Microsoft SDKs\Azure\AzCopy\AzCopy.exe"))
{
    throw "Azcopy not installed - get it from here: https://azure.microsoft.com/en-gb/documentation/articles/storage-use-azcopy/"
}

Write-host ""
Write-Host "Starting backup for account" -ForegroundColor Yellow 
Write-host "--------------------------" -ForegroundColor Yellow
Write-Host " -Account: $Account"
Write-Host " -Token: $SASToken"

$response = Invoke-WebRequest -Uri "https://$Account.table.core.windows.net/Tables/$SASToken"
[xml]$tables = $response.Content
$tableNames = $tables.feed.entry.content.properties.TableName

Write-host ""
Write-host "Found Tables to backup" -ForegroundColor Yellow
Write-host "--------------------------" -ForegroundColor Yellow
foreach ($tableName in $tableNames)
{
    Write-Host " -Table: $tableName"
}


foreach ($tableName in $tableNames)
{
    $url = "https://$Account.table.core.windows.net/$tableName"

    Write-host ""
    Write-Host "Backing up Table: $url"-ForegroundColor Yellow
    Write-host "--------------------------" -ForegroundColor Yellow
    Write-host ""

    & "${env:ProgramFiles(x86)}\Microsoft SDKs\Azure\AzCopy\AzCopy.exe" /Source:$url /Dest:$OutputDir\$account\ /SourceSAS:$SASToken /Z:"$env:temp\$([guid]::NewGuid()).azcopyJournal"   

    Write-host ""
    Write-host "Backup completed" -ForegroundColor Green
    Write-host ""
    Write-host ""

}

For more details on usage have a look here:

https://gripdev.wordpress.com/2016/04/08/backup-azure-table-storage-quick-powershell-script/

Answer 3

You can backup any Azure Table Storage table (not blobs though) with free software like Slazure Light. The following C# code backup all your Azure Tables to json files:

Download NuGet packages first:

Install-Package Azure.Storage.Slazure.Light

Create a console application in Visual Studio and add the following code:

using System;
using System.Linq;
using Microsoft.WindowsAzure.Storage.Table;
using Newtonsoft.Json;
using SysSurge.Slazure.AzureTableStorage;

namespace BackupAzureTableStore
{
    class Program
    {
        /// <summary>
        /// Usage: BackupAzureTableStore.exe "UseDevelopmentStorage=true"
        /// </summary>
        /// <param name="args"></param>
        static void Main(string[] args)
        {
            var storage = new DynStorage(args.Length == 0 ? "UseDevelopmentStorage=true" : args[0]);

            foreach (var cloudTable in storage.Tables)
            {
                var tableName = cloudTable.Name;
                var fileName = $"{tableName}.json";

                using (var file = new System.IO.StreamWriter(fileName))
                {
                    var dynTable = new DynTable(storage.StorageAccount, tableName);

                    TableContinuationToken token = null; // Continuation token required if > 1,000 rows per table
                    do
                    {
                        var queryResult =
                            dynTable.TableClient.GetTableReference(tableName)
                                .ExecuteQuerySegmented(new TableQuery(), token);
                        file.WriteLine("{{{0} : [", JsonConvert.SerializeObject(tableName));
                        var cntr = 0;
                        foreach (var entity in queryResult.Results)
                        {
                            var dynEntity = dynTable.Entity(entity.PartitionKey, entity.RowKey);
                            dynEntity.LoadAll().ToList(); // Force pre-downloading of all properties

                            file.WriteLine("{0}{1}", cntr++ > 0 ? "," : string.Empty,
                                JsonConvert.SerializeObject(dynEntity));
                        }

                        file.WriteLine("]}");

                        token = queryResult.ContinuationToken;
                    } while (token != null);
                }

            }

            Console.WriteLine("Done. Press a key...");
            Console.ReadKey();
        }
    }
}