Submit spark app (Spark on AKS) from Azure Data Factory

0
votes

We have deployed Apache Spark on Azure Kubernetes Services (AKS).

Able to submit spark application via CLI https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode

Question: Is it possible to submit a spark job/run a spark application from Azure Data factory version 2? That way we can orchestrate spark application from data factory.

1
apache-sparkkubernetesazure-data-factoryazure-aks

1 Answers

0
votes
High-Level Architecture

enter image description here

Quick explanation of the architecture flow:

  • In order to connect any on-premise data sources to Azure, you can install an integration runtime (executable installer from Data Factory) on a dedicated VM. This allows Data Factory to create connections to these on-premise servers.

  • A Data Factory pipeline will load raw data into Data Lake Storage Gen 2. ADLS2 applies hierarchical namespace to blob storage (think folders). A downstream task in the pipeline will trigger a custom Azure Function.

  • Custom python Azure Function will load a config yaml from ADLS2, and submit a spark application to k8s service via k8s python client. The container registry is essentially a docker hub in Azure. Docker images are deployed to the registry, and k8s will pull as required.

  • Upon submission by the Azure Function, k8s will pull the spark image from container registry and execute the spark application. Spark leverages the k8s scheduler to automatically spin up driver and executor pods. Once the application is complete, the executor pods self-terminate while the driver pod persists logs and remains in "completed" state (which uses no resources).

ADF Pipeline Example:

enter image description here

few challenges with this setup:

  1. No programmatic way of submitting a spark application to k8s (running command line "kubectl" or "spark-submit" isn't going to cut it in production)
  2. No OOTB method to orchestrate a spark application submission to k8s using Data Factory
  3. Spark 2.4.5 with Hadoop 2.7 doesn't support read/writes to ADLS2, and cannot be built with Hadoop 3.2.1 (unless you have extensive dev knowledge of spark source code)

Let's walk through the top secret tools used to make the magic happen. Deployed a custom spark-on-k8s-operator resource to the kubernetes cluster which allows submitting spark applications with a yaml file. However, the documentation only shows how to submit a spark app using cmd line kubectl apply -f yaml. To submit the spark application programmatically (ie - via REST API), leveraged the CustomObjectsApi from k8s python client SDK inside a python Azure Function. Why? Because ADF has an OOTB task to trigger Azure Functions. 🎉

Spark 3.0.0-preview2 already has built-in integration with Hadoop 3.2+ so that's not so top secret, but there are a couple of things to look out for when you build the docker image. The bin/docker-image-tool.sh needs extra quotes on line 153 (I think this is just a problem with windows filesystem). To support read/write to ADLS2, you need to download the hadoop-azure jar & wildfly.openssl jar (place them in spark_home/jars). Finally, replace the kubernetes/dockerfiles/spark/entrypoint.sh with the one from Spark 2.4.5 pre-built for Hadoop 2.7+ (missing logic to support python driver).

Quick tips: package any custom jars into spark_home/jars before building your docker image & reference them as dependencies via "local:///opt/spark/jars", upload extra python libs to ADLS2 and use sc.addPyFile(public_url_of_python_lib) in your main application before importing.

Reference: https://www.linkedin.com/pulse/ultimate-evolution-spark-data-pipelines-azure-kubernetes-kenny-bui/