CERTIFICATE_VERIFY_FAILED via spark-submit vs successful manual execution of python code

Question

python 2.7, aws emr 5.29 cluster bootstrapped with boto 3 on all it's nodes.

Success : ssh & execute a python program that inserts into dynamodb (tested on both master & a core node). executed manually via python test.py

Failure : spark-submit --jars test.py (on master) Error : From the stacktrace it is pointing to CERTIFICATE_VERIFY_FAILED to the dynamodb aws url.

Question: what is spark-submit doing under the covers that requires authentication that straight python test.py did not require authentication?

Answer 1

After adding ca bundle to aws config as mentioned here and re-ran via spark-submit CERTIFICATE_VERIFY_FAILED went away.