Azure Data Factory: can I pause a Pipeline run while another Pipeline is running and then resume the first Pipeline?

0
votes

So I have this Pipeline that runs for a long time (weeks), which loads some tables with Data Factory and processes them with Databricks.

Also, I have another Pipeline that is run each day for a couple of hours. However, the Databricks cluster seems not to be powerful enough to run both pipelines simultaneously, as it throws an error when both Pipelines are active (seems to be a memory error, "Spark driver has stopped unexpectedly").

The daily Pipeline is highest priority, though, so ideally I would like to pause for around 3 hours the long term Pipeline, then execute the daily trigger, and then resume the long term Pipeline execution.

Is it possible to do that?

Thanks in advance!

2
azureazure-pipelinesazure-data-factoryazure-data-factory-2azure-databricks
I believe the short answer is 'No', you can't pause a running a pipeline. Is there a reason both processes have to run in the same DataBricks cluster? I would have to query the architecture of a single pipeline running for weeks, but in the meantime you could investigate running the jobs in separate clusters. - Joel Cochran
@JoelCochran well, the reason is that cluster management is not something we are in charge of, and the capacity that we currently have doesn't allow for a second cluster. I might be able to ask for bigger capacity to run another cluster, but that's something I was trying to avoid if possible. - PoweToor

2 Answers

0
votes

The simple answer: No. You should look for the solution in the databricks cluster throwing the error. There is no way to pause a pipeline because another pipeline in running.

0
votes

You cannot pause a running pipeline. Azure devops pipeline doesnot have this feature currently. Below screenshots lists all the statuses that a build could be. As you can see "pause" is not one of them.

enter image description here

You might have to ask for a bigger capacity to run another cluster.

However, you can always submit a feature request(Click Suggest a feature and choose Azure Devops) to Microsoft Development team. Hope they will consider implementing this feature in the future sprints.

enter image description here