Slow performance when copying from HTTP source to blob sink

Question

I use a copy activity to call an HTTP API and store the json response as a file in Azure blob storage. The copy activity is executed in a ForEach loop and each activity run takes 16 seconds, but when I look at the run details it says the copy duration is only 3 seconds. Then why does the activity take 16 seconds to complete? The source dataset is an Http File with an HttpServer linked service and the sink dataset is a blob storage json file. Both the source and sink datasets are configured with Binary Copy and it's a GET request to an HTTPS URL with anonymous authentication.

I would like to speed up this acticity since it is run multiple times inside the ForEach loop. Is there some way to improve the performance?

Answer 1

There is always a few seconds of overhead when starting an activity. Also consider that the http server might be also responsible for some of those seconds you are seeing there.

If you are using a for each loop and want to speed up the process, you can uncheck the Sequential check in the settings tab of the foreach activity.

Hope this helped!