Add meta data to a data lake file using ADF

Question

Azure Data Factory v2 has a Get Metadata activity which can read meta data on the files stored in ADLS. It can preserve the meta data on files when it moves/copies the files. But is there a way to add or modify meta data on the lake files using ADF?

Answer 1

The meta data are created by Data Lake(Storage) once the files are uploaded on.

These properties can not be changed unless you delete and re-upload them to Data Lake(or Stroage). Some others have asked the same questions about how to change this meta data in Stack overflow. You could easily find these by seraching.

But if you modify the content of the file in Data Lake, such as add or delete columns, the size, columnCount and structure can be changed.

So for the question "is there a way to add or modify meta data on the lake files using ADF?", the answer is no, there isn't.

HTP.

Answer 2

Yes there's a way.

You can make use of Azure Blob Storage API:

set-blob-metadata method for Blob Storage

Data lake is just an extension to underlying Blob Storage engine

So, you can hook up a web activity in your pipeline and call the rest api pointing at your blob and it will set metadata for you.