Azure Queue Storage: Send files in messages

Question

I am assessing Azure Queue Storage to communicate between two decoupled applications.

My requirement is to send a file (flat file, size: small to large) in the queue message.

As per my reading an individual message in a queue cannot exceed beyond 64KB, so sending a file of variable size in the message is out of question.

Another solution I can think of is using a combination of Queue Storage and blob storage, i.e. in the queue message add a reference to the file (on blob storage) and then when required read the file from the blob (using the reference/address in the queue message).

My question is, is this a right approach? or are there any other elegant ways to achieving this?

Thanks, Sandeep

Answer 1

While there's no right approach, since you can put anything you want in a queue message (within size limits), consider this: If your file sizes can go over 64K, you simply cannot store these within a queue message, so you will have no other choice but to store your content somewhere else (e.g. blobs). For files under 64K, you'll need to decide whether you want two different methods for dealing with files, or just use blobs as your file source across the board and have a consistent approach.

Also remember that message-passing will eat up bandwidth and processing. If you store your files in queue messages, you'll need to account for this with high-volume message-passing, and you'll also need to extract your file content from your queue messages.

One more thing: If you store content in blobs, you can use any number of tools to manipulate these files, and your files remain in blob storage permanently (until you explicitly delete them). Queue messages must be deleted after processing, giving you no option to keep your file around. This is probably an important aspect to consider.