Azure Data Factory - Python Custom Activity

Question

I am trying to create a data factory using Python Custom Activity (similar to .Net) to extract the data from source files and do some pre-processing on them. After the data is pre-processed, need to upload the file to a blob. I have a python code ready to do this but wanted to explore if i can use Data Factory Custom Activity to do this by injecting Python code into it. If yes, please tell me how to do it, links or any reference docs would help.

Appreciate your thoughts/ideas on this.

Answer 1

Dependencies should be installed at batch pool (referenced by custom activity) level. There are two ways to install dependencies on batch pool: use pool startup scripts docs.microsoft.com/en-us/azure/batch/… or custom image docs.microsoft.com/en-us/azure/batch/batch-custom-images

Answer 2

You can use ADF V2 custom activity for that, here's the document https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-dotnet-custom-activity