UnitTest for .NET Apache Spark

0
votes

I want to write unit tests for my Spark Application written in C#/.NET. I'm currently using XUnit for writing tests but I haven't found any good documentation for writing unit tests to test my spark application components.

I have written a spark unit tests but it fails if I run "dotnet test" command. If I run my tests using spark-submit there is no output like test passed or failed. Spark Submit command I used is "spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin/Debug/netcoreapp3.1/microsoft-spark-2.4.x-0.12.1.jar dotnet bin/Debug/netcoreapp3.1/MedicalClaims.SparkUnitTests.dll"

Any help will be very much appreciated.

1
unit-testingapache-sparkxunit.net.net-spark

1 Answers

0
votes

I fixed and the tests actually work. I was missing

test in spark submit. So the correct statement for running your spark test with C#/.NET is spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin/Debug/netcoreapp3.1/microsoft-spark-2.4.x-0.12.1.jar dotnet test bin/Debug/netcoreapp3.1/MedicalClaims.SparkUnitTests.dll