How to setup seemly circular but not actually circular dependencies in Azure Data Factory V2

1
votes

Below is the scenario :

Pipeline A triggered by Tumbling Window Trigger A - Runs Every Hour.

Pipeline B triggered by Tumbling Windows Trigger B - Runs Every Week.

I want A to run only after :

a. last run of A is successful. (self dependency - doable).

b. last run of B is successful. ( This is also doable but can get into circular dependency because of point b in below lines)

I want B to run only after :

a. last run of B is successful. ( self dependency - doable)

b. last run of A is successful. ( This gets into circular dependency issue when we publish and later when it generates ARM templates)

Here A is having frequency of 1 hour and B is having frequency of 1 week.

consider B is to next execute on 9th Feb 12:00 AM for window of 2nd Feb - 9th Feb.

Also, A is about to execute on 9th Feb 12:00 Am for window of 8th Feb 11:00 PM - 12:00 PM.

In this case both runs will collide at 9th Feb 12:00 AM. Can we setup dependency so that it does not collide.

1
azureazure-data-factoryazure-data-factory-2
hi,does my answer helps you? – Jay Gong
Hi, i am yet to try this approach. But i guess, an idea that i can use from below is , to setup dependency via Execute pipeline dependency and make a facade pipeline rather than doing similar thing via triggers, to avoid circular dependencies. – Divyesh
Yes,i agree that trigger is needed.Any concern about your test,pls let me know., – Jay Gong

1 Answers

1
votes

I provide below ideas for your reference:

1.Firstly,put the pipelines into Execute Pipeline Activity because you may have to do some steps before and after execution of A and B pipelines.

2.Secondly,since the pipelines have to be executed with some conditions,i think you could persist the results of A and B pipelines execution anyway.For example,after executing pipeline,use an Azure Function Activity or Web Activity to send the result of pipeline into some residences. The aim is logging the latest execute result of A and B.

3.Finally,before the execution of A and B pipelines,you could use an Until Activity to evaluate whether the pipeline could be executed.