Is it possible to store variables in Azure Data Factory pipelines?

2
votes

In my Azure Data Factory pipeline, I want to use a variable, which gets updated on each run and which is also read on each run. At the moment, I am using a Database to achieve that. But it would be much simpler if Azure Data Factory provided a way of storing variables. So, my question is, is there any such facility in Azure Data Factory?

1
azureazure-data-factoryazure-data-factory-2
Do you mean Set Variable activity cannot meet your request(Variable update logic ) ? – Joseph Xu
@HarrisXu Can the value be passed to the next Pipeline run? – MetallicPriest
You can declare a parameter in PipelineB, when you executing PipelineB in PipelineA, you can pass the value form PipelineA variable to PipelineB parameter. – Joseph Xu
I want to use that variable for the same pipeline runs. Let us say the pipeline runs every 10 minutes. I want the pipeline in one iteration to persist the value so that it can be used in the next iteration. – MetallicPriest
They can't persist inside ADF across pipeline runs. If you really need something persistent, you're going to need to write data to a storage mechanism and then read it in subsequent pipelines. IMO, @Harris-Xu recommended the best approach. – Joel Cochran

1 Answers

1
votes

As @Joel Cochran says, ADF doesn't support persist a variable inside pipeline runs. We need to write data to a storage, eg. database or azure storage. Use Lookup Activity to get the value from blob storage file or DB. :)