I have two separate Data Factories on my Azure Subscription, lets call them DF-A and the other DF-B In Data Factory DF-A I have a pipeline and when this has completed, I would like the Pipeline on DF-B to run; how would I achieve this?
Thanks
0
votes
In ADF it can not achieve this. Do you consider using logic app? Logic app can achieve this.
- Joseph Xu
Do you have an example
- viperdenwo82
Did either of our answers help?
- GregGalloway
Hi @viperdenwo82, If my answer is helpful for you, you can accept it as answer( click on the check mark beside the answer to toggle it from greyed out to filled in.). This can be beneficial to other community members. Thank you.
- Joseph Xu
2 Answers
0
votes
In Logic app designer, you can create two pipeline run steps to trigger the two pipelines in different Data Factory running.
It is more easier by using logic apps to achieve this.
- create a Recurrence trigger to schedule the executions and two Azure Data Factory operations to trigger the pipeline running.
- In the Azure Data Factory operations, select Create a pipeline run Action.
- The summary is here:
0
votes
While it's possible, it's much more complicated than one pipeline executing another from within the same Azure Data Factory.
In DF-A create a pipeline called ExecuteExternalPipeline copying the following JSON into the Code tab:
{
"name": "ExecuteExternalPipeline",
"properties": {
"description": "Executes an ADF pipeline in a different ADF",
"activities": [
{
"name": "StartPipelineThenWait",
"description": "Calls the ADF REST API to start a pipeline in another ADF running using the MSI of this current ADF. Then it waits on a webhook callback",
"type": "WebHook",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"url": {
"value": "@concat(\n 'https://management.azure.com/subscriptions/',\n pipeline().parameters.SubscriptionID,\n '/resourceGroups/',pipeline().parameters.ResourceGroup,\n '/providers/Microsoft.DataFactory/factories/',\n pipeline().parameters.DataFactory,\n '/pipelines/',\n pipeline().parameters.Pipeline,\n '/createRun?api-version=2018-06-01'\n)",
"type": "Expression"
},
"method": "POST",
"body": {
"value": "@json(\n concat(\n '{\n \"InputFileName\": \"', pipeline().parameters.InputFileName, '\"\n }'\n )\n)\n",
"type": "Expression"
},
"timeout": "20:00:00",
"authentication": {
"type": "MSI",
"resource": "https://management.azure.com"
}
}
},
{
"name": "ThrowErrorIfFailure",
"type": "IfCondition",
"dependsOn": [
{
"activity": "StartPipelineThenWait",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"expression": {
"value": "@if(equals(activity('StartPipelineThenWait').status,'success'),true,json('throw an error!'))",
"type": "Expression"
}
}
}
],
"parameters": {
"SubscriptionID": {
"type": "string",
"defaultValue": "12345abcd-468e-472a-9761-9da416b14c0d"
},
"ResourceGroup": {
"type": "string",
"defaultValue": "DF-B-RG"
},
"DataFactory": {
"type": "string",
"defaultValue": "DF-B"
},
"Pipeline": {
"type": "string",
"defaultValue": "ChildPipeline"
},
"InputFileName": {
"type": "string",
"defaultValue": "File1.txt"
}
},
"annotations": []
}
}
Then create ChildPipeline in DF-B with the following code:
{
"name": "ChildPipeline",
"properties": {
"activities": [
{
"name": "DoYourLogicHere",
"description": "",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "https://google.com",
"type": "Expression"
},
"method": "GET"
}
},
{
"name": "CallbackSuccess",
"description": "Do not remove this activity. It notifies the process which executed this pipeline that the pipeline is complete.",
"type": "WebActivity",
"dependsOn": [
{
"activity": "DoYourLogicHere",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@pipeline().parameters.callBackUri",
"type": "Expression"
},
"method": "POST",
"body": {
"value": "@json(concat('{\"status\": \"success\", \"pipelineRunId\": \"',pipeline().RunId,'\"}'))",
"type": "Expression"
}
}
},
{
"name": "CallbackFail",
"description": "Do not remove this activity. It notifies the process which executed this pipeline that the pipeline is complete.",
"type": "WebActivity",
"dependsOn": [
{
"activity": "DoYourLogicHere",
"dependencyConditions": [
"Failed"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@pipeline().parameters.callBackUri",
"type": "Expression"
},
"method": "POST",
"body": {
"value": "@json(concat('{\"status\": \"failure\", \"pipelineRunId\": \"',pipeline().RunId,'\"}'))",
"type": "Expression"
}
}
}
],
"parameters": {
"callBackUri": {
"type": "string",
"defaultValue": "https://google.com"
},
"InputFileName": {
"type": "string",
"defaultValue": "File1.txt"
}
},
"annotations": []
}
}
Replace the DoYourLogicHere activity with your own activities but leave the two callback activities.
Then you need to find the MSI (see the Properties tab of your DF-A in the Azure Portal) for DF-A and make it a Data Factory Contributor on DF-B so that it can execute the pipeline in the other ADF.
