Azure Data Factory Limits

0
votes

I have created a simple pipeline that operates as such:

  1. Generates an access token via an Azure Function. No problem.

  2. Uses a Lookup activity to create a table to iterate through the rows (4 columns by 0.5M rows). No problem.

  3. For Each activity (sequential off, batch-size = 10):
  4. (within For Each): Set some variables for checking important values.
  5. (within For Each): Pass values through web activity to return a json.
  6. (within For Each): Copy Data activity mapping parts of the json to the sink-dataset (postgres).

Problem: The pipeline slows to a crawl after approximately 1000 entries/inserts.

I was looking at this documentation regarding the limits of ADF.

  • ForEach items: 100,000
  • ForEach parallelism: 20

I would expect that this falls within in those limits unless I'm misunderstanding it. I also cloned the pipeline and tried it by offsetting the query in one, and it tops out at 2018 entries.

Anyone with more experience be able to give me some idea of what is going on here?

1
azure-data-factory-2

1 Answers

2
votes

As a suggestion, whenever I have to fiddle with variables inside a foreach, I made a new pipeline for the foreach process, and call it from within the foreach. That way I make sure that the variables get their own context for each iteration of the foreach.

Have you already checked that the bottleneck is not at the source or sink? If the database or web service is under some stress, then going sequential may help if your scenario allows that.

Hope this helped!