How to run .Net spark jobs on Databricks from Azure Data Factory?

1
votes

In Azure data factory, you have a Databricks Acvitiy. This activity supports running python, jar and notebooks. And These notebooks may be written in scala, python, java, and R but not c#/.net.

Is there inherent or direct support where I can write my .NET spark code and run it on Databricks from Data Factory?

Can I use .NET spark in Azure Databricks to its full extent?

1
c#azure-data-factoryazure-databricks.net-spark
Have you referred to docs.microsoft.com/en-us/dotnet/spark/tutorials/…? – Jim Xu
Yes I have. But as you can see this is customised and uses CLI. As I mentioned I have also done the same. What I want to know is that in any way can I invoke .NET spark on bricks using Azure data factory – Nikhil Sharma

1 Answers

1
votes

You specify that you want to launch a JAR file. The .NET for Apache Spark project uses a JAR file to start a listener which the .NET code then connects to.

The JAR is microsoft-spark-2.4.x-0.12.1.jar (depends on the version of spark and the .NET NuGet version). The Class to run is org.apache.spark.deploy.dotnet.DotnetRunner which you need to pass the correct parameters to, so that it starts your .NET application.

Ed