Azure Data Factory v2 Pipeline API call for trigger timestamp

Question

I am new to working with azure data factory and I am trying to set up a pipeline that fetches data from an API for a specific time frame. Here is what I have been doing so far: I created a tumbling window pipeline for API call to bring in data as JSON file and store in ADL folder with specific timestamp. The problem is that I want just data for the current day and hour and store in that specific folder e.g.year/month/day/hour/ json file How can I tell the URL just to fetch data within the hour the trigger was activated (trigger is set for every full hour and needs to fetch data from e.g. 3am till 3:59. I was able to get data from specific timestamp but not current time which would be linked to trigger time. URL code: ?sourceId=1LuOA,9VKZ2,CISG1,D7UIQ,gu4me,hSZGT,K582n,MnkPP,Mxgt4,N4hAZ,PvECt,Qhr1i,uIWnW,vot1K,XAbJ4,XRH1E,ZbKW0,Zjrs3,ZttLo&interval=S10&maxResult=500&startTimestamp=2018-05-16T00:00:00%2B00:00&endTimestamp=2018-05-16T00:59:59%2B00:00

sink Code:

beacon/@{formatDateTime(adddays(utcnow(),0),'yyyy')}/@{formatDateTime(adddays(utcnow(),0),'MM')}/@{formatDateTime(adddays(utcnow(),0),'dd')}

when I try to add hours I get a weird folder that does not represent hours.

Answer 1

In your pipeline, define a pipeline parameter, say windowStart. Then Pass trigger().outputs.windowStartTime to your pipeline parameter. And then if your expected folder name is abc/year/month/day/hour. You could useabc/@{formatDateTime(pipeline().parameters.windowStart,'yyyy')/@{formatDateTime(pipeline().parameters.windowStart,'MM')/@{formatDateTime(pipeline().parameters.windowStart, 'dd')/@{formatDateTime(pipeline().parameters.windowStart,'HH')}

Please reference this for tumbling window. https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger#tumbling-window-trigger-type-properties