How to pass several parameters to a function which is iterated by executor.map() from concurrent.futures.ProcessPoolExecutor

Question

The issue was posted several times here in stackoverflow, but none of them could help me as my case is very specific.

I have a function Myfunction(p1,p2,p3,p4) which need 4 parameters to run.

I need to run Myfunction on multiprocessing way.

I use concurrent.futures.ProcessPoolExecutor to do this job.

I know how to pass a list as an argument in the executor.map(Myfunction, argument)

But this time, I have a list of tuples which is the list of my 4 arguments for my multiprocessing.

Here is my code:

def Myfunction(p_udid,p_systemPort,p_deviceName, p_version, p_os):
    desired_caps = {}
    desired_caps['platformName'] = p_os
    desired_caps['platformVersion'] = p_version
    desired_caps['deviceName'] = p_deviceName
    desired_caps['udid'] = p_udid
    desired_caps['noReset'] = 'true'



if __name__ == '__main__':
    list=[('41492968379078','4730','S6S5IN3G','6','Android'),('53519716736397','4731','S6S5IN3G','6','Android'),('0123456789ABCDEF','4732','20','7','Android')]
    with concurrent.futures.ProcessPoolExecutor() as executor:
        multiprocesses = executor.map(Myfunction, list)

Of course I get an error :

concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py", line 239, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py", line 198, in _process_chunk return [fn(*args) for args in chunk] File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py", line 198, in return [fn(*args) for args in chunk] TypeError: Myfunction() missing 4 required positional arguments: 'p_systemPort', 'p_deviceName', 'p_version', and 'p_os' """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "E:/DropboxBACKUP14112018/Cff/Python/project_GITHUB/test2.py", line 24, in for multiprocess in multiprocesses: File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists for element in iterable: File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py", line 598, in result_iterator yield fs.pop().result() File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py", line 435, in result return self.__get_result() File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py", line 384, in __get_result raise self._exception TypeError: Myfunction() missing 4 required positional arguments: 'p_systemPort', 'p_deviceName', 'p_version', and 'p_os

I tried different stuff from all the answers similar to my issue, but I didn't succeed.

Can someone help me please?

Thanks @stovfl it doesn't show anymore error message, but my function is not running. As you guessed, I tried to run some smartphone automation with appium & python. So I tested by hand the function : Myfunction('53519716736397','4731','S6S5IN3G','6','Android') Myfunction('41492968379078','4730','S6S5IN3G','6','Android') And it works. But when I tried to run it with multiprocessing: multiprocesses = executor.map(Myfunction, *list) I didn't get error message, but I didn't see the function runing the test on smartphones. Do you have any idea why? — Gauthier Buttez
idea why?: Don't use *list, note the *, add a print(args) at top of Myfunction(*args) — stovfl
Ok. it prints the args... it is passing the each tuple as 1 arg. Look what it shows the print(args): (('41492968379078', '4730', 'S6S5IN3G', '6', 'Android'),) (('53519716736397', '4731', 'S6S5IN3G', '6', 'Android'),) (('0123456789ABCDEF', '4732', '20', '7', 'Android'),). So that means it is passing the list of tuples as arguments, instead of passing the items of 1 tuple. — Gauthier Buttez
I have a list of tuple, executor.map need to pass 1 tuple of 5 items as 5arguments. But for the moment it is passing 1 list of 4 tuples (which contains 5 items). SO as a result, Myfunction receive 3 tuples of 5 items as arguments. — Gauthier Buttez

YOLO YOLO · Accepted Answer · 2020-01-31T12:24:16

Here's a way you can make it work:

def Myfunction(*args):

    p_udid,p_systemPort,p_deviceName, p_version, p_os = args[0]

    desired_caps = {}
    desired_caps['platformName'] = p_os
    desired_caps['platformVersion'] = p_version
    desired_caps['deviceName'] = p_deviceName
    desired_caps['udid'] = p_udid
    desired_caps['noReset'] = 'true'
    return desired_caps

def cpu_tasks(func, *args):

    # set chunksize to be even 
    with ProcessPoolExecutor() as tp:
        result = tp.map(func, chunksize=10, *args)
    return list(result)

if __name__ == '__main__':
    lst=[('41492968379078','4730','S6S5IN3G','6','Android'),('53519716736397','4731','S6S5IN3G','6','Android'),('0123456789ABCDEF','4732','20','7','Android')]
    ans = cpu_tasks(Myfunction, *[lst])

    print(ans)

[{'platformName': 'Android',
  'platformVersion': '6',
  'deviceName': 'S6S5IN3G',
  'udid': '41492968379078',
  'noReset': 'true'},
 {'platformName': 'Android',
  'platformVersion': '6',
  'deviceName': 'S6S5IN3G',
  'udid': '53519716736397',
  'noReset': 'true'},
 {'platformName': 'Android',
  'platformVersion': '7',
  'deviceName': '20',
  'udid': '0123456789ABCDEF',
  'noReset': 'true'}]

How to pass several parameters to a function which is iterated by executor.map() from concurrent.futures.ProcessPoolExecutor

1 Answers