Azure Table Storage RowKey restricted Character Patterns?

Question

Are there restricted character patterns within Azure TableStorage RowKeys? I've not been able to find any documented via numerous searches. However, I'm getting behavior that implies such in some performance testing.

I've got some odd behavior with RowKeys consisting on random characters (the test driver does prevent the restricted characters (/ \ # ?) plus blocking single quotes from occurring in the RowKey). The result is I've got a RowKey that will insert fine into the table, but cannot be queried (the result is InvalidInput). For example:

RowKey: 9}5O0J=5Z,4,D,{!IKPE,~M]%54+9G0ZQ&G34!G+

Attempting to query by this RowKwy (equality) will result in an error (both within our app, using Azure Storage Explorer, and Cloud Storage Studio 2). I took a look at the request being sent via Fiddler:

GET /foo()?$filter=RowKey%20eq%20'9%7D5O0J=5Z,4,D,%7B!IKPE,~M%5D%54+9G0ZQ&G34!G+' HTTP/1.1

It appears the %54 in the RowKey is not escaped in the filter. Interestingly, I get similar behavior for batch requests to table storage with URIs in the batch XML that include this RowKey. I've also seen similar behavior for RowKeys with embedded double quotes, though I have not isolated that pattern yet.

Has anyone co me across this behavior? I can easily restrict additional characters from occurring in RowKeys, but would really like to know the 'rules'.

Answer 1

The following characters are not allowed in PartitionKey and RowKey fields:

• The forward slash (/) character
• The backslash (\) character
• The number sign (#) character
• The question mark (?) character

Further Reading: Azure Docs > Understanding the Table service data model

Answer 2

public static readonly Regex DisallowedCharsInTableKeys = new Regex(@"[\\\\#%+/?\u0000-\u001F\u007F-\u009F]");

Detection of Invalid Table Partition and Row Keys:

bool invalidKey = DisallowedCharsInTableKeys.IsMatch(tableKey);

Sanitizing the Invalid Partition or Row Key:

string sanitizedKey = DisallowedCharsInTableKeys.Replace(tableKey, disallowedCharReplacement);

At this stage you may also want to prefix the sanitized key (Partition Key or Row Key) with the hash of the original key to avoid false collisions of different invalid keys having the same sanitized value.

Do not use the string.GetHashCode() though since it may produce different hash code for the same string and shall not be used to identify uniqueness and shall not be persisted.

I use SHA256: https://msdn.microsoft.com/en-us/library/s02tk69a(v=vs.110).aspx

to create the byte array hash of the invalid key, convert the byte array to hex string and prefix the sanitized table key with that.

Also see related MSDN Documentation: https://msdn.microsoft.com/en-us/library/azure/dd179338.aspx

Related Section from the link: Characters Disallowed in Key Fields

The following characters are not allowed in values for the PartitionKey and RowKey properties:

The forward slash (/) character

The backslash (\) character

The number sign (#) character

The question mark (?) character

Control characters from U+0000 to U+001F, including:

• The horizontal tab (\t) character

• The linefeed (\n) character

• The carriage return (\r) character

Control characters from U+007F to U+009F

Note that in addition to the mentioned chars in the MSDN article, I also added the % char to the pattern since I saw in a few places where people mention it being problematic. I guess some of this also depends on the language and the tech you are using to access the table storage.

If you detect additional problematic chars in your case, then you can add those to the regex pattern, nothing else needs to change.

Answer 3

I just found out (the hard way) that the '+' sign is allowed, but not possible to query in PartitionKey.

Answer 4

I found that in addition to the characters listed in Igorek's answer, these also can cause problems (e.g. inserts will fail):

• |
• []
• {}
• <>
• $^&

Tested with the Azure Node.js SDK.

Answer 5

In addition to the above you can't use an underscore _ at the start of a partition key either, an error is returned that a key starting with an underscore is not a valid partition key.

Answer 6

I transform the key using this function:

private static string EncodeKey(string key)
{
    return HttpUtility.UrlEncode(key);
}

This needs to be done for the insert and for the retrieve of course.