Azure DataFactory Copy Data - how to know it has done copying data?

Question

We have a bunch of files in azure blob storage as a tsv format and we want to move them to destination which is ADLS Gen 2 and parquet format. We want this activity on daily basis. So the ADF pipeline will write bunch of parquet files in folders which will have date in them. for example

../../YYYYMMDD/*.parquet

On the other side we have API which will access this. How does the API know that the data migration is completed for a particular day or not?

Basically is there an in built ADF feature to write done file or _SUCCESS file which API can rely on?

Thanks

Answer 1

Why not simply call the API to let it know from ADF using Web activity?

You can use Web Activity to even pass the name of processed file as URL or body parameters to that the API knows what to process.

Answer 2

Provide two ways here for you.Let me say,from the perspective of ADF copy activity execution results, it can be divided into active way and passive way.

1.Active way,you could use waitOnCompletion feature in execute pipeline activity.

After that,execute a web activity to trigger your custom api.Please see this case:Azure Data Factory: How to trigger a pipeline after another pipeline completed successfully

2.Passive way,you could use monitor feature of ADF pipeline. Please see the example of .net sdk:

Console.WriteLine("Checking copy activity run details...");

RunFilterParameters filterParams = new RunFilterParameters(
    DateTime.UtcNow.AddMinutes(-10), DateTime.UtcNow.AddMinutes(10));
ActivityRunsQueryResponse queryResponse = client.ActivityRuns.QueryByPipelineRun(
    resourceGroup, dataFactoryName, runResponse.RunId, filterParams);
if (pipelineRun.Status == "Succeeded")
    Console.WriteLine(queryResponse.Value.First().Output);
else
    Console.WriteLine(queryResponse.Value.First().Error);
Console.WriteLine("\nPress any key to exit...");
Console.ReadKey();

Check the status is successed, then do your custom business.