ADF Delta Copy Activity fails, delete partly copied file from data lake

0
votes

I have a hourly delta load pipeline in that i have a copy activity to copy data from sql server to datalake parquet format with folder structure (YY/MM/DD/table_nameHH) in For each activity:

after copy activity i have success/failure procedure to update records in control table

enter image description here

i need to ensure if any copy activity failed in between i need to remove that partly copied file from data lake. how to put that condition to pickand delete that dynamic file in my pipeline.

Thanks in Advance

p.s i am pretty new to this tool and learning daily.

1
azureazure-data-factoryazure-data-factory-2azure-data-lake-gen2
Hi @Kumar AK, hope you're doing well. If my answer is helpful for you, hope you can accept it as answer. This can be beneficial to other community members. Thank you. - Leon Yue
Do you have any other concerns? - Leon Yue
Thanks Leon for your input. You said if condition should behind the copy activity? then how can we get @equals(activity('Copy_Dimcustomer_AC').output.executionDetails.status,'Succeeded') if we add copy activity befor copy? do you mean do i need to add after copy? if i add copy after then during the copy if it's fail then our if condition will not work for failure cases right? - Kumar AK
Hi @Kumar AK, that's the output of the copy active, we can get the copy active status from the output JSON string. According the status value, we could know if the copy active succeeded or failed. I made the example according my understanding of your screenshots. In each inner active, it's a flow actually. If if condition true, run the true active flow, false to false active flow. You could get more details from the if condition document. - Leon Yue
Please let me know if it's helpful for you, thank you~ - Leon Yue

1 Answers

1
votes

Please ref these steps bellow:

You could add an If condition behind the Copy_DimCustomer_AC active: enter image description here

In If condition expressions, build the expression to judge if Copy_DimCustomer_AC.executionDetails.status equals "Succeeded", if ture, then the copy active succeeded:

@equals(activity('Copy_Dimcustomer_AC').output.executionDetails.status,'Succeeded')

enter image description here

True actives:

enter image description here

False active: add a delete active to delete the file and run the Log_failure_status_AC:

enter image description here

HTH.