How to get just certain data from azure blob storage in Azure Data Factory performing copy activity with destination as SQL database?

0
votes

I am trying to perform Copy Activity inside Azure Data Factory where the source is Blob and destination is MySql database.

I do not want certain data to be copied during that activity and that can be determined by reading First key value Element of the Blob JSON.

Or maybe if it is not possible to stop reading all the BLOB data then atleast while writing inside the SQL table I want just some specific data.

I saw that there is Pre-copy Script in the copy activity. Can someone give an idea how to use it or what kind of script I can write there?

1
sqlazureetlazure-blob-storageazure-data-factory

1 Answers

0
votes

The pre copy script doesn't work for your requirement. you can use dataflow with a filter transformation to cleanse your data.

enter image description here

The doc is https://docs.microsoft.com/en-us/azure/data-factory/data-flow-create