Run .Net and Python applications on Azure Data Factory?

Question

I am trying to create a pipeline in Azure which involves running .Net applications and python applications as part of the workflow. These applications read/write to ADLS and Azure databases. I am wondering what are the best tools for the job.

So far I have landed on using ADF, with Azure Batch/Custom activity for .Net; and same or Databricks notebook/Python activity for Python as per this. (preferably avoid having to spin up a Databricks cluster.)

Other options I considered: use Azure Functions for .Net (and use it in ADF) but not sure if it is only suitable for small snippets of code or entier applications. Use Logic Apps to run .net/python applications in a Docker container activity step which seems like an overkill.

Thanks in advance.

Answer 1

Per my knowledge, maybe Azure Function is not suitable for entire application. Based on the statement in Azure Function Overview, Functions is a great solution for processing data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and microservices. It is more suitable for snippet of code to process data in schedule.

For both .Net application and Python application, you could use custom activity. Please see the Github guidance and this Python tutorial.