Azure-data-Factory Copy data If a certain file exists

Question

I have many files in a blob container. However I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container. I used Get metadata and IF Condition on Data Factory. Can you please help me with the dynamic script for this. I tried this @bool(startswith( activity('Get Metadata1').output.childitems.ItemName, 'SRManifest.csv')). It doesnt work.

Then I thought, what if i used @greaterOREquals(activity('Get Metadata1').output.LastModified,adddays(utcnow(),-2))But this checks the last modified within 2 days of the Bloob not the file exist. Thank you.

Please see below my diagram

Answer 1

I have understood your requirement differently I think.

I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container

1 Change your metadata activity to look for existence of sentinel file (SRManifest.csv)

2 Follow with an IF activity, use this condition:

3 Put your sp in the True part of the IF activity

If you also needed the file list passed to the sp then you'll need the GetMetadata with childitems option inside the IF-True activity

Answer 2

Based on your diagram, since you are looping over all the blob names already, you can add a Boolean variable to the pipeline and set its default value to false:

Inside the ForEach activity, you only want to attempt to set the variable if the value is still false, and if the blob name is found, set it to true. Since Set Variable cannot be self-referential, do this inside the False branch of an If activity:

This will only attempt to process if the value is false (so the file name has not been found yet), and will do nothing if the value is true. Now set the variable based on your file name:

[NOTE: This value can be hard coded, parameterized, or based on a variable]

When you execute the pipeline, you'll see the Set Variable stops attempting once the value is set to true:

In the main pipeline, after the ForEach activity has completed, you can use the variable to set the condition of your final If activity. If the blob is never found, it will still be false, so put the Stored Procedure activity inside the True branch.