Azure Data Factory V2 Check file size for Copy Activity

Question

I need to run a copy activity for files in an on-premises data source. I need to check the file sizes and copy them if the size is greater than 0. If the file's size is 0, I will send a notification so that the admin would check the source.

The problem is, in Get Metadata activity, the arguments in the field list do not include size. Also, Validation activity, does not look for size property either. Any suggestion? Again, the files live on-premises.

I added If Condition after the Get Metadata activity and used the expression:

@greaterOrEquals(activity('Get-metadata-source').output.size,0)

Which generates the following error:

The expression 'greaterOrEquals(activity('Get-metadata-source').output.size,0)' cannot be evaluated because property 'size' doesn't exist, available properties are 'childItems, effectiveIntegrationRuntime, executionDuration, durationInQueue, billingReference'.

Thank you!

Only the file as dataset in Get Metadata can choose the argument: size. Folder doesn't support it. And even the dataset file is set from parameter, it still not work. Base on that, We can't achieve your request. — Leon Yue
Thanks Leon. Can I use a wildcard in the file field in the file path in the dataset? I tried it against one file and was able to compare by size. But not sure how to do it for multiple files. Thanks again — elasticSol
Hi @elasticSol, I made the example for you, please see my answer. — Leon Yue
Hello @elasticSol, do you have any other concerns? If my answer is helpful for you, hope you can accept it as answer. This can be beneficial to other community members. Thank you. — Leon Yue
Hi @LeonYue Thank you very much. I didn't have the chance to finish it but I'm working on it this week. I appreciate it!! — elasticSol

Leon Yue Leon Yue · Accepted Answer · 2020-12-06T15:04:26

If you choose a file not a folder as dataset in Get Metadata active, you can get the file size.

If you files are in the same folder or location, you can follow the steps bellow:

Use one Get Metadata to get the list of filenames: ARGUMENT: Child items

Use For Each active to loop all the files:@activity('Get Metadata1').output.childitems:

In Foreach active, create an another dataset 2 has the same path with dataset 1， set the file name with parameter.

In Foreach active, use another Get Metadata to get the file size. Pass the foreach item.name to the filename:

Run the pipeline and get the file size.

You could add the if condition to judge the file size and build your pipeline.

HTP

Azure Data Factory V2 Check file size for Copy Activity

1 Answers