Paginate REST API call in Data Factory

Question

I want to call a ZenDesk REST API in Data Factory, here's how the output looks like

next_page determines the next URL page of the REST API

end_of_stream returns false if I've reached the end of the page

In Data Factory, I've set up a Copy Activity in a pipeline which copies the data from REST JSON into my Azure DB, something like this

The problem is the pagination in ADF doesn't support the type of pagination that Zendesk API provides, from my research it looks like I'll need to create an Until loop in order to make this work, so something like this

I need to know how I can

• Set a variable to true/false based on the Copy Activity output from the REST API call
• Dynamically set the URL of my Copy Activity, so it can loop through all the pages in the REST API

Thanks!

Answer 1

Set a variable to true/false based on the Copy Activity output from the REST API call

In the value Variables section of the Set Variable Activity, in the variable click on the add dynamic content.

@activity('Copy data1').output.end_of_stream

You can replace the Copy data1 with that of your own Copy Activity Name

Dynamically set the URL of my Copy Activity, so it can loop through all the pages in the REST API

For your REST API dataset,

Configure the below Parameter

Relative URL :

In the Copy Data Activity :

Once the first copy action is done, you should update the variable start_time with the value of the end_time of the output.

Snippet Output of the REST API

The reason being, the next page url is same relative url of the API with start_time parameter with the value of end_time of the REST API (Current Page) Output.