Setting up the Start time in Azure Data Factory pipeline

Question

Currently I am working on setting up the pipeline for one of the data export. My expectation from pipeline scheduler is : - The pipeline scheduler should get triggered on 00:00 AM PST time everyday - Currently my start & end time configuration is as below :

     "start": "2017-10-10T07:00:00Z",
     "end": "2017-10-12T07:00:00Z"

• As per my understanding, I have just adjusted the UTC time slice by +7 hrs so that it will trigger my pipeline on the same day 00:00 AM PST time.

Can anyone let me know whether I am following the correct approach? or is there other approach i should take?

Thanks,

Answer 1

Yes, that's correct. You can also use Delay/offset option for any pipeline which you want to start after a specific time.

Answer 2

I assume your start and end are on the activity. This is not a schedule for when to run the processing on a regular basis, but an active window for when the data factory should process. It is the start and end of the full execution window, not each individual slice execution.

You have two options to achieve what you are after, documented here and summarised below. both of the below introduce a 6 hour offset.

1) Set the offset on the upstream data set:

"availability": {
        "frequency": "Day",
        "interval": 1,
        "offset": "06:00:00"
    },

2) Set the delay on the pipeline:

"policy": {
          "timeout": "1.00:00:00",
          "delay": "06:00:00",
          "concurrency": 1,
          "executionPriorityOrder": "NewestFirst",
          "style": "StartOfInterval",
          "retry": 3,
          "longRetry": 0,
          "longRetryInterval": "00:00:00"
    },

Also look at controlling behaviour with StartOfInterval and EndOfInterval depending when you want the activity to execute within the scheduled period. See the docs here for further info.