How to include blob metadata in copy data mapping

2
votes

I'm working on a ADF v2 pipeline, which copies data from csv blob to Azure SQL database table. For each load I would like to collect source metadata, like source blob name, and save it to a target table as a part of data lineage framework.

My blob source run the following schema:

  1. StoreName,
  2. StoreLocation,
  3. StoreTaxId.

My destination table run the following schema:

  1. StoreName,
  2. StoreLocation,
  3. DwhProcessDate,
  4. DwhSourceName.

I do not know, how to properly include name of the source in the mapping section of Copy Data activity.

For the moment I have:

  1. defined a [Get Metadata1] activity to get references to all blobs that are available from Azure Blob Storage
  2. defined a [ForEach1] activity, iterating through the output of an expression @activity('Get Metadata1').output.childitems
  3. inside the [ForEach1] activity, I have placed [Copy Data1] activity, where I have source and sink sections defined.

What I'm looking for is a way to add extra line to the mapping section, which will samehow bind @item().name to destination column [DwhSourceName]

Thanks for all suggestion on how to achieve this.

1
azureazure-data-factory
So,does my answer helps you? – Jay Gong

1 Answers

0
votes

Actually,based on my test,you can specify the dymatic content of column key,but you can't set blob metadata as value of columns in copy data mapping at the pipeline run time. Please see the rules mentioned in this document.

enter image description here

You still need to add the FileName column in your source data before the copy activity.Maybe you could use Azure Blob Trigger Function to get the blob file name so hat you could add the FileName column when any data stream into the blob.(Please refer to this case:How Do I get the Name of The inputBlob That Triggered My Azure Function With Python)