I am creating a test Azure Data Factory Pipeline for learning purposes.
For this specific pipeline, I want to move files from one blob to the other. In ADF world, this involves creating a pipeline with the following activities:
- Get Metadata: retrieve file list from Blob 1
- Filter: from the output of previous activity, filter out folders, and output list of files
- Move files: get output from previous activity, move files from Blob 1 to Blob 2
ADF does not implement a Move File activity, so activity 2. above is actually a ForEach activity composed by two sub-activities:
2.1 Copy Files: Copy file from Blob 1 to Blob 2 2.2 Delete Files:Delete file from Blob 1
When debugging this pipeline, I am getting the following error for activity 2.2:
Failed to execute delete activity with data source 'AzureBlobStorage' and error 'The required Blob is missing. Folder path: sensor-sink-aws/test - Copy.json/.'
But the file exists in the container:
In my DeleteFile activity, I have the following data for Source:
Which basically references the Source Dataset that I created for this pipeline - it is a reusable dataset, meaning that I'm able to pass the container name and file name in a dynamic way.
Any idea on what could be going wrong?