Java heap space error while converting pandas dataframe to spark dataframe

Question

Below is the defined spark_session:

spk = SparkSession \
    .builder \
    .master("yarn") \
    .config("spark.submit.deployMode","client") \
    .config("spark.sql.broadcastTimeout", "10000000") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")\
    .appName("pp_test") \
    .config("spark.sql.shuffle.partitions", "1000") \
    .config("spark.driver.maxResultSize", "5g") \
    .config("spark.sql.execution.arrow.enabled", "true") \
    .config("spark.sql.sources.bucketing.enabled", "true") \
    .config("spark.sql.inMemoryColumnarStorage.compressed", "true") \
    .config("spark.sql.inMemoryColumnarStorage.batchSize", 100000) \
    .config("spark.sql.files.maxPartitionBytes", 41943040) \
    .config("spark.executor.heartbeatInterval", "10000000")\
    .config("spark.network.timeout", "10000000")\
    .config("spark.io.compression.codec", "snappy")\
    .config("spark.sql.orc.filterPushdown", "true") \
    .config("spark.yarn.queue", "public") \
    .config("spark.sql.sources.partitionOverwriteMode","dynamic") \
    .config("spark.yarn.dist.archives","hdfs:///user/grp_gdoop/anaconda/anaconda2_env.zip#ANACONDA") \
    .enableHiveSupport() \
    .config("hive.exec.dynamic.partition", "true") \
    .config("hive.exec.dynamic.partition.mode", "nonstrict") \
    .config("hive.enforce.bucketing", "false") \
    .config("hive.enforce.sorting", "false") \
    .getOrCreate()

I have a pandas DF with shape: 3592761, 5 (3592761 rows & 5 columns): named pdf

sdf = spk.createDataFrame(pdf)

The above code throws the below error.

   ``` An error occurred while calling  
   z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.
    : java.lang.OutOfMemoryError: Java heap space
    ```

I further sub divided the above pandas dataframe: pdf to 40 sub pandas dataframe with 89820 rows each and it fails to convert to pyspark dataframe.

Any guidance on what i might be missing in the session config would be super helpful!