Setting executor environment variables in Spark

1
votes

I am using spark-defaults classification [http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html] and providing the following property:- spark.yarn.appMasterEnv.FOO = "foo" to set environment variables in the yarn master.

I wanted to know if these variables will be present in the executor instances as well? If not, is there a way to do the same?

1
apache-sparkenvironment-variablesyarnamazon-data-pipeline

1 Answers

3
votes

Environment variables set by property spark.yarn.appMasterEnv.FOO = "foo" won't be present on executor instances.

I use the below Classification to set-up environment variables on executor and master node:

[   
  {
   "Classification": "yarn-env",
   "Properties": {},
   "Configurations": [
       {
         "Classification": "export",
         "Properties": {
             "VARIABLE_NAME": VARIABLE_VALUE,
         }
       }
   ]
 }
]