How to cache data between Azure Durable Function orchestration instances?

0
votes

Documentation states that Azure Durable Function orchestrations code should be deterministic, cos of replays. In my case, I have some data in Azure Table Storage, that I need to fetch in workflow. The workflow is recursive and the data in Azure Table Storage can change during execution, and it is OK to have stale state for ~1 min. In regular code I would rely on memory cache to improve the performance. But in orchestrations, suppose it can not be used directly, cos this makes workflow non-deterministic.

I can still use cache in activity and call it from orchestrations, but every activity call involves serialization\deserialization of inputs\outputs and passing messages though control queue. These operations are heavier then fetching data itself.

So I have a question, is there any pattern, that can be used to cache data between orchestration instances in memory, without wrapping this logic in activity?

2
azureazure-functionsazure-durable-functions

2 Answers

2
votes

What I can suggest you is: use a distributed cache, specifically Redis Cache for Azure.

I drew an image for you:

enter image description here

Get your data from Azure Table Storage in your orchestration, do your operation in there and save it to Redis cache. Then pass the id of the required data to each activity. Then you can get the data from Redis cache inside each activity.

This is a solution with cache as you asked. However, please note that if you want high-performance data query, Azure Table Storage is not the best solution to work with. I suggest you to use either Azure SQL or CosmosDB. But if you are seeking a cheap option that's fine. But in that case, Redis cache won't be good option for you, because it's not a cheap solution neither. If this Redis cache won't work for you, I would suggest you review your algorithm.

Good luck!