Process multiple files in azure blob storage simultaneously based on custom logic

2
votes

My exact requirement is like this:

I have bunch of files for my different customers in my blob storage "B1". Lets say 10 files for customer "C1", 10 files for customer "C2" and 10 files for customer "C3".

I want to perform some operation on each file and move them to blob storage "B2". This can take 30 seconds to 5 minutes depending on data in file.

Now I want to process one files for each customer simultaneously but not more then one file for the same customer at the same time.

Like one file for customer "C1", one for "C2" and one for "C3" must process simultaneously. So the processing time of "C1" do not effect "C2" and "C3". But the next file of C1 will be processed only when the first is completed.

What is the best architecture with Microsoft Azure functionalities to implement this?

For example, I have implemented like this with Azure Function V1:

  1. Blob Triggered Azure Function : This will simply add file names with customer ID in an azure table as soon as any file is placed in blob. This table will contain one more column "InQueue" which is FALSE by default.

  2. Time Triggered Azure Function : This will check in azure table and take first file for each customer for whom all files has InQueue = FALSE (Means : No file in process). And for them update InQueue = TRUE and add their name to azure queue.

  3. Queue Triggered Azure Function : This will be triggered as soon as any file is in azure queue and do the process on it. Once the process is completed it will delete the entry for that file from azure table. So, now for the customer of that file all other entries has "InQueue" = FALSE (No file in process)

So in above architecture Time Triggered azure function is taking care of one file per customer but it is also pushing multiple files of different customers in queue. And as Queue Triggered azure function can run multiple instance at the same time. All files of different customers will be executed simultaneously.

Is my architecture is good? or bad? or how can I improve it? What are the other options that can make my process faster or easier or with less steps?

1
azureazure-functionsazure-functions-runtimeazure-functions-core-tools
Please give some comment on this. – Yash

1 Answers

1
votes

Now the main problem confusing you what I understand is you want to execute the multiple function simultaneously. If so, I suggest you trying the Logic App with parallel branches.

This is the tutorial about how to create a logic app with parallel branch. And this is the rendering. And you could add Azure Functions as a action. enter image description here

Here I used a Recurrence(Time Schedule) as a trigger you could use others. And after each branch you could also add actions to it.Just like the pic shows. enter image description here

Hope this could help you, if you still have other questions, please let me know.