Arrays in Azure Table Storage Entities (Non-Byte)

Question

I'm looking to store arrays in Azure Table entities. At present, the only type of array supported natively is byte-array, limited to 64k length. The size is enough, but I'd like to store arrays of longs, doubles and timestamps in an entity.

I can obviously cast multiple bytes to the requested type myself, but I was wondering if there's any best-practice to achieve that.

To clarify, these are fixed length arrays (e.g. 1000 cells) associated with a single key.

Answer 1

I have written a Azure table storage client, called Lucifure Stash, which supports arrays, enums, large data, serialization, public and private properties and fields and more.

You can get it at https://github.com/hocho/LucifureStash

Answer 2

I've been trying to think of a nice way to do this other than the method you've already mentioned, and I'm at a loss. The simplest solution I can come up with is to take the array, binary serialize it and store in a binary array property.

Other options I've come up with but dismissed:

1. If storing it natively is important, you could keep this information in another child table (I know Azure Tables don't technically have relationships, but that doesn't mean you can't represent this type of thing). The downside of this being that it will be considerably slower than your original.
2. Take the array, XML serialize it and store it in a string property. This would mean that you could see the contents of your array when using 3rd party data explorer tools and you could run (inefficient) queries that look for an exact match on the contents of the array.
3. Use Lokad Cloud fat entities to store your data. This essentially takes you're whole object, binary serializes it and splits the results into 64kb blocks across the properties of the table entity. This does solve problems like the one you're experiencing, but you will only be able to access your data using tools that support this framework.

Answer 3

If you have just a key-value collection to store, then you can also check out Azure BLOBs. They can rather efficiently store arrays of up to 25M time-value points per single blob (with a random access within the dataset).

Answer 4

If you choose to store your object in blob storage and need more than one "key" to get it, you can just create an azure table or two or n where you store the key you want to look up and the reference to the exact blob item.

Answer 5

Why don't you store the values as csv strings?

Answer 6

You could serialize your array as a JSON string using the .NET JavaScript serializer: http://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer.aspx

This class has a "MaxJsonLength" property you could use to ensure your arrays didn't exceed 64K when you were serializing them. And you can use the same class to deserialize your stored objects.