Azure data factory - delivered files are not being recognised

Question

I'm using azure data factory with blob storage as an external source (the data table is marked as external).

I've recently restored a data source that had not been providing data, however I'm noticing that it does not recognise that the files are now in the blob store. There file/folder namining conventions have not changed.

Each of my slices for my first pipeline are all marked with the below status:

Waiting: Dataset dependencies

What should I look at to debug this issue? Is there any way to force data factory to re-check for missing files? (the assumption seems to be it will do this automatically, but it isn't picking the files up - I've also tried re-marking the slices to process, but this hasn't helped).

Answer 1

Obviously, your external datasets are waiting for the next execution (to enter to window slice you defined). If you restored all the blobs you needed, then you can go to that pipeline in Diagram and click it, so it opens running list Pipeline runs you can right-click any of these and then click Run, after your external data is ready and you change its availabitlity:)

Answer 2

This is because the external dataset has failed, having not detected the file when it was first expected to arrive (or on the subsequent X retries as defined in the dataset definition).

Once all of the retry attempts are exhausted, the file isn't checked again, so if it arrives after this point, the pipeline won't execute. As this is in the dataset and not the pipeline, rerunning the pipeline will not work. The only method I know to correct this is to open the azure portal (not monitor and manage) navigate to the "recently failed datasets" blade and hit rerun on the corresponding entry.