Azure Data Factory V2 Copy Activity to Data Warehouse from Blob storage

1
votes

I'm using ADF v2 to import data from CSV source on Blob Storage to Azure SQL Dat Warehouse. My first Activity in the Pipeline is Copy activity from CSV (on hot blob storage) to Azure SQL DWH. If this activity finish successfully the next one is again Copy activity but now from CSV (on hot blob storage) to CSV (on cool blob storage), move process for history and storage pricing optimization. Filename of the CSV file is sent to Pipeline as Pipeline-Parameter.

What I noticed is that if I start the Pipeline with Filename parameter of the csv that does not exists on the blob storage, first Copy activity (CSV -> DWH) finish without error. It does not import any data to DWH, which is normal, but raise no error of missing file. Next Copy activity (CSV hot storage -> CSV cool storage) raise Failed error -> blob is missing.

I would like also Copy to DWH to raise Error, because that way I can add other activity in the pipeline that will handle this situation. Log error somewhere, mark file as missing, etc.

What is specific in Copy activity from Storage to DWH that is finishing successfully without source blob existence? I use polybase checkbox in the Sink part of the activity if that counts for something.

Does anyone knows how to set this Copy activity (Storage CSV -> SQL DWH) to Fail if source blob does not exists?

Thanks

1
azure-data-factory-2
Hi Darko. Maybe some solution can be to use adf getmetadata activity and then check with if condition if that file exists, if exist you continue with your process. For logging, I also have similar scenarios where I want to log errors/info but for now, didn't come to an ideal solution. Also, I don't think they will go deeper in some logical stuff with ADF, in my talk with Microsoft guy, he told me, and i quote: "and the general idea is that ADF is the orchestration engine rather than doing much work itself". – DraganB
Hi Dragan. Thanks for your reply. Your solutions looks ok, I will try to use If condition activity to resolve this. Can I point from If activity to existing activities in the pipeline ? Thanks – Darko

1 Answers

0
votes

yes. You can put as many as you want activities in your if activity. So you can take details about blob storage with getMetadata activity (check exists property in the documentation, link below).

https://docs.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity