How can I create a one time pipeline run with a long delay in c# data factory

Question

I need to trigger a pipeline I have built inside of my azure data factory with certain parameters based off of a file I have stored in a database. My problem is that I need to schedule this pipeline to trigger ONCE after a certain amount of time( will usually be hours). This is needed for scheduling and I can't do it event driven. I am using the .NET SDK

I have already created a connection to my data factory and created a schedule trigger. My problem is that a schedule trigger doesn't allow me to trigger one time and then stopping. It requires intervals and a stop date, I tried to set the stop date the same as the start date but it gives me the error of "interval cannot exceed end date".

 for (int x = 0; x < intervals.Count; x++)
            {
                // Create a schedule trigger
                string triggerName = location + deliveryDate+x;
                ScheduleTrigger myTrigger = new ScheduleTrigger()
                {

                    Pipelines = new List<TriggerPipelineReference>()
                    {
                        // Associate the Adfv2QuickStartPipeline pipeline with the trigger
                        new TriggerPipelineReference()
                        {
                            PipelineReference = new PipelineReference(pipelineName),
                            Parameters = pipelineParameters,
                        }
                    },
                    Recurrence = new ScheduleTriggerRecurrence()
                    {
                        StartTime = intervals[x],
                        TimeZone = "UTC",
                        EndTime = intervals[x],
                        Frequency = RecurrenceFrequency.Day

                    }





                };

                // Now, create the trigger by invoking the CreateOrUpdate method
                triggerResources.Add(triggerName,new TriggerResource()
                {
                    Properties = myTrigger
                });
            }

I cannot do a pipeline run because there is not way for me to do a run after a certain delay (like 2 hours) if this was possible I would just create a delayed pipeline run...I have tried everything like leaving the frequency blank, changing it to every possibility, and even using different trigger classes like tumbling and event.

Have you considered writing a script (i.e. cron job) on your local machine, and leaving on overnight, which will Invoke the pipeline manually for you? — MartinJaffer-MSFT
@MartinJaffer-MSFT This is a solution that would work, but it's not the solution that I need. We are server-less and really want to stay that way, which is also the whole reason I use the DataFactory in the first place because of its orchestration abilities. — DanielKBoyer

MartinJaffer-MSFT MartinJaffer-MSFT · Accepted Answer · 2019-08-09T21:26:43

There is a simple, crude solution. Create a new pipeline with a parameter of integer type. The first activity in the pipeline will be a Wait Activity. Use the parameter to set how long the Wait Activity should last. The Second activity in the pipeline will be an Execute Pipeline Activity, which depends on the Wait Activity, and will trigger the pipeline you really want to run.

This solution lets you choose how long to wait, then execute the real pipeline you want to run. The Wait Activity is in seconds, I think, so you will need to do some arithmetic. However since you can trigger manually, it shouldn't be a problem.

How can I create a one time pipeline run with a long delay in c# data factory

1 Answers