Copy and Extracting Zipped XML files from HTTP Link Source to Azure Blob Storage using Azure Data Factory

Question

I am trying to establish an Azure Data Factory copy data pipeline. The source is an open HTTP Linked Source (Url reference: https://clinicaltrials.gov/AllPublicXML.zip). So basically the source contains a zipped folder having many XML files. I want to unzip and save the extracted XML files in Azure Blob Storage using Azure Data Factory. I was trying to follow the configurations mentioned here: How to decompress a zip file in Azure Data Factory v2 but I am getting the following error:

ErrorCode=UserErrorSourceNotSeekable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Your HttpServer source can't support random read which is requied by current copy activity setting, please create two copy activities to work around it: the first copy activity binary copy your HttpServer source to a staging file store(like Azure Blob, Azure Data Lake, File, etc.), second copy activity copy from the staged file store to your destination with current settings.,Source=Microsoft.DataTransfer.ClientLibrary,'

Not exactly sure what is going wrong, but it would be really helpful if someone can guide me with the procedure.

Answer 1

I broke this up in to two Copy data activities in order to separate the donwloading of the zip file (which is quite large) and the unpacking. You could try and do them in one step but I think you're going to run into timeout issues. With my approach you also have a copy of the original zip file which would be good for audit trail and debugging purposes.

I try and document my ADF patterns in a boxes and lines format which shows the key details for each component. So here there are two Copy activities, and the supporting linked services and datasets - try and follow this, let me know how you get on:

NB it took quite a long time for ADF to unpack the .xml files as there are rather a lot of them. My results showing in Azure Storage Explorer: