reading config file in python with ConfigParser

Question

I have used configparser to read configurations in my python written program . I am reading file from s3 for now , but my requirement is to configurations defined in the program itself rather than from any other external source . code written is below:

config = configparser.ConfigParser()
config.readfp(open(s3:path\config))

config file format : config.ini

[section1]
var1=Y
var2=Y
var3=['col1','col2']

I am reading above file that is located in s3 , but instead of reading from s3 I want to read from program in itself . what needs to be done in order to achieve this ?

above code is written in pyspark program , I am passing config file with spark submit command but to read config file i need to provide path and this is not desirable . spark submit in aws emr :

'Args': ['spark-submit','--deploy-mode', 'cluster','--master', 'yarn','--executor-memory', conf['emr_step_executor_memory'],'--executor-cores',  conf['emr_step_executor_cores'],'--conf','spark.yarn.submit.waitAppCompletion=true','--conf','spark.rpc.message.maxSize=1024',f'{s3_path}/file1.py',  '--py-files',f'{s3_path}/file2.py',f'{s3_path}/file3.py',f'{s3_path}/file4.py','--files', f'{s3_path}/config ]

because of config.readfp(open(s3:path\config)) line i need to provide s3 path , that is not desirable options are either pass config file from spark submit and make available to every other python files those are reading configs or read configuration inside of program itself .

The whole purpose of having a config file is to separate runtime configuration values from the code. If you don't want to do that, use configparser to write the config to a temporary location and pass that to Spark. — BoarGules

f4him f4him · Accepted Answer · 2020-12-18T05:52:50

you can do something like this:

parser = configparser.ConfigParser()
parser.read_dict({'section1': {'key1': 'value1',
                                'key2': 'value2',
                                'key3': 'value3'},
                   'section2': {'keyA': 'valueA',
                                'keyB': 'valueB',
                                'keyC': 'valueC'},
                   'section3': {'foo': 'x',
                                'bar': 'y',
                                'baz': 'z'}})

Check this for detailed info as well as some other alternative way to achieve this.

Sample retrieval

parser["section1"]["key1"]
'value1'

reading config file in python with ConfigParser

1 Answers