Spark execution on kubernetes - driver pod fails

Question

I try to run simple spark code on kubernetes cluster using spark 2.3 native kubernetes deployment feature.

I have a kubernetes cluster running. At this time, the spark code does not read or write data. It creates an RDD from list and print out the result, just to validate the ability to run kubernetes on spark. Also, copied the spark app jar in the kubernetes container image too.

Below is the command i run.

bin/spark-submit --master k8s://https://k8-master --deploy-mode cluster --name sparkapp --class com.sparrkonk8.rdd.MockWordCount --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=myapp/sparkapp:1.0.0 local:///SparkApp.jar

2018-03-06 10:31:28 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: sparkapp-6e475a6ae18d3b7a89ca2b5f6ae7aae4-driver namespace: default labels: spark-app-selector -> spark-9649dd66e9a946d989e2136d342ef249, spark-role -> driver pod uid: 6d3e98cf-2153-11e8-85af-1204f474c8d2 creation time: 2018-03-06T15:31:23Z service account name: default volumes: default-token-vwxvr node name: 192-168-1-1.myapp.engg.com start time: 2018-03-06T15:31:23Z container images: dockerhub.com/myapp/sparkapp:1.0.0 phase: Failed status: [ContainerStatus(containerID=docker://3617a400e4604600d5fcc69df396facafbb2d9cd485a63bc324c1406e72f0d35, image=dockerhub.com/myapp/sparkapp:1.0.0, imageID=docker-pullable://dockerhub.com/sparkapp@sha256:f051d86384422dff3e8c8a97db823de8e62af3ea88678da4beea3f58cdb924e5, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://3617a400e4604600d5fcc69df396facafbb2d9cd485a63bc324c1406e72f0d35, exitCode=1, finishedAt=Time(time=2018-03-06T15:31:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-03-06T15:31:24Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 2018-03-06 10:31:28 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses:

Container name: spark-kubernetes-driver Container image: myapp/sparkapp:1.0.0 Container state: Terminated Exit code: 1

I guess you already to run this code on your local machine, right? Do you have more logs? kubectl logs maybe? — TobiSH
I know this isn't an answer (but I am not allowed to leave a comment), but I'd suggest you look at the logs: > kubectl logs sparkapp-6e475a6ae18d3b7a89ca2b5f6ae7aae4-driver and paste the output here. — joshuarobinson
Did you solve this issue? I have the same issue, I tried on GCP with building from Spark source code. All attempts failed. While the master node gave me the similar info like yours, the probe with kubectl describe only has created container successfully message as the last message with no more update. I don't know where to look at now. — LiweiZ

Maverick Maverick · Accepted Answer · 2018-03-06T17:14:40

Below is the spark config that driver pod submits. I pulled this from K8s UI. @TobiSH let me know if this helps for troubleshooting my issue.

SPARK_DRIVER_MEMORY: 1g
SPARK_DRIVER_CLASS: com.sparrkonk8.rdd.MockWordCount
SPARK_DRIVER_ARGS: 
SPARK_DRIVER_BIND_ADDRESS: 
SPARK_MOUNTED_CLASSPATH: /SparkApp.jar:/SparkApp.jar
SPARK_JAVA_OPT_0: -Dspark.kubernetes.executor.podNamePrefix=sparkapp-028d46fa109e309b8dfe1a4eceb46b61
SPARK_JAVA_OPT_1: -Dspark.app.name=sparkapp
SPARK_JAVA_OPT_2: -Dspark.kubernetes.driver.pod.name=sparkapp-028d46fa109e309b8dfe1a4eceb46b61-driver
SPARK_JAVA_OPT_3: -Dspark.executor.instances=5
SPARK_JAVA_OPT_4: -Dspark.submit.deployMode=cluster
SPARK_JAVA_OPT_5: -Dspark.driver.blockManager.port=7079
SPARK_JAVA_OPT_6: -Dspark.kubernetes.container.image=docker.com/myapp/sparkapp:1.0.0
SPARK_JAVA_OPT_7: -Dspark.app.id=spark-5e3beb5109174f40a84635b786789c30
SPARK_JAVA_OPT_8: -Dspark.master= k8s://https://k8-master
SPARK_JAVA_OPT_9: -Dspark.driver.host=sparkapp-028d46fa109e309b8dfe1a4eceb46b61-driver-svc.default.svc
SPARK_JAVA_OPT_10: -Dspark.jars=/opt/spark/work-dir/SparkApp.jar,/opt/spark/work-dir/SparkApp.jar
SPARK_JAVA_OPT_11: -Dspark.driver.port=7078`

Spark execution on kubernetes - driver pod fails

2 Answers