Suppose, we have parameter pipeline in Azure Data Factory which copy data from Dataset_DL_XXX (Data Lake) to Dataset_DB_XXX (Azure Database). this activity starts at 10:00 pm. At 10:05 pm we have 1 more request to copy data from Dataset_DL_YYY (Data Lake) to Dataset_DB_YYY (Azure Database).
Can we use the same pipeline to trigger multiple tasks? or do we have any other approach?
Thank You.
Can we use the same pipeline to trigger multiple tasks?
Based on my researching, it should be possible if you use control flow in the pipeline.My idea as below:
1.Firstly, please see the statements from this doc:
A pipeline run in Azure Data Factory defines an instance of a pipeline execution. For example, say you have a pipeline that executes at 8:00 AM, 9:00 AM, and 10:00 AM. In this case, there are three separate runs of the pipeline, or pipeline runs. Each pipeline run has a unique pipeline run ID. A run ID is a GUID that uniquely defines that particular pipeline run.
So you could trigger the same pipeline at 10 pm and 10.05 pm.Then are different pipeline runs.
2.Get the trigger time inside the pipeline:@pipeline().TriggerTime
3.Use If-Condition Activity to control that which copy activity should be executed this time.
