What is the recommended way to use HDInsight cluster for running pipeline for custom activity in Azure Data Factory. Can we use single HDInsght cluster for multiple Azure Data Factory jobs and for multiple pipelines running simultaneously?
1 Answers
2
votes
If you are referring to running a custom .NET activity using Azure Data Factory, you would run it just like a HDInsight activity, just labeling as a custom activity and linking to the .NET dll zip file in blob storage. Running with Azure Batch is also an option, as the .NET will work well and Azure Batch is cheaper (if the custom activity is the only reason for having a HDInsight cluster).
You are able to use a single HDInsight cluster to run multiple data factory jobs, including multiple parallel pipelines. However, keep in mind that depending on your parallel jobs (number, size, etc.) and configuration of your cluster you may overload the cluster's resources.
