Python genetic optimisation multiprocessing with a global constant variable, how to speed up?

Question

I am writing a genetic optimization algorithm based on the deap package in python 2.7 (goal is to migrate to python 3 soon). As it is a pretty heavy process, some parts of the optimisation are processed using the multiprocessing package. Here is a summary outline of my program:

Configurations are read in and saved in a config object
Some additional pre-computations are made and saved as well in the config object
The optimisation starts (population is initialized randomly and mutations, crossover is applied to find a better solution) and some parts of it (evaluation function) are executed in multiprocessing
The results are saved

For the evaluation function, we need to have access to some parts of the config object (which after phase 2 stays a constant). Therefore we make it accessible to the different cores using a global (constant) variable:

from deap import base
import multiprocessing

toolbox = base.Toolbox()

def evaluate(ind):
    # compute evaluation using config object
    return(obj1,obj2)

toolbox.register('evaluate',evaluate)

def init_pool_global_vars(self, _config):
    global config
    config = _config

...
# setting up multiprocessing
pool = multiprocessing.Pool(processes=72, initializer=self.init_pool_global_vars,
                                        initargs=[config])
toolbox.register('map', pool.map_async)
...
while tic < max_time:
    # creating new individuals
    # computing in optimisation the objective function on the different individuals
    jobs = toolbox.map(toolbox.evaluate, ind)
    fits = jobs.get()
    # keeping best individuals

We basically make different iterations (big for loop) until a maximum time is reached. I have noticed that if I make the config object bigger (i.e. add big attributes to it, like a big numpy array) even if the code is still same it runs much slower (fewer iterations for the same timespan). So I thought I would make a specific config_multiprocessing object that contains only the attributes needed in the multiprocessing part and pass that as a global variable, but when I run it on 3 cores it is slower than with the big config object and on 72 cores, it is slightly faster, but not much.

What should I do in order to make sure my loops don't suffer in speed from the config object or from any other data manipulations I make before launching the multiprocessing loops?

Running in a Linux docker image on a linux VM in the cloud.

No I'm running it in a Linux docker image on a linux VM in the cloud — Thomas
But I cannot do this as if I would want to call the multiprocessing.Pool at the start, I don't have yet my config object created, as such I cannot initialize the different executors with the object as a global variable (as at the beginning, it does not exist yet or not complete yet) — Thomas
All sub-processes created by Pool will inherit the state (memory) from the parent process which gets pickled for transfer. Hence if you make your config object bigger, more pickling needs to be done and more memory needs to copied. Since the sub-processes inherit the global state anyway, it's not necessary to use that extra initializer function. When you create your specific config_multiprocessing, supposedly derived from config, did you del config before spawning the pool? Because if not then both objects config_multiprocessing and config will be copied to the sub-processes. — a_guest
This is not completely true as when I don't pass config to the multiprocessing pool (i.e. pool = multiprocessing.Pool(processes=config.nProcesses)) I have an error saying that config is not defined NameError: global name 'config' is not defined I am running inside a linux docker container so there the sub-process created by pool does not inherit the state of the parent! — Thomas

hume hume · Accepted Answer · 2020-03-16T18:33:38

The joblib package is designed to handle cases where you have large numpy arrays to distribute to workers with shared memory. This is especially useful if you are treating the data in shared memory as "read-only" like what you describe in your scenario. You can also create writable shared memory as described in the docs.

Your code might look something like:


import os

import numpy as np
from joblib import Parallel, delayed
from joblib import dump, load

folder = './joblib_memmap'
try:
    os.mkdir(folder)
except FileExistsError:
    pass

def evaluate(ind, data):
    # compute evaluation using shared memory data
    return(obj1, obj2)

# just used to initialize memory mapped data
def init_memmap_data(original_data):
    data_filename_memmap = os.path.join(folder, 'data_memmap')
    dump(original_data, data_filename_memmap)
    shared_data = load(data_filename_memmap, mmap_mode='r')
    return shared_data

...
# however you set up indices needs to be changed here
indexes = range(10)  

# however you load your numpy data needs to be done here
shared_data = init_memmap_data(numpy_array_to_share)  

# change n_jobs as appropriate
results = Parallel(n_jobs=2)(delayed(evaluate)(ind, shared_data) for ind in indexes)  

# get index of the maximum as the "best" individual
best_fit_individual = indexes[results.argmax()]

Additionally, joblib supports a threading backend that may be faster than the process based one. It will be easy to test both with joblib.

Python genetic optimisation multiprocessing with a global constant variable, how to speed up?

1 Answers