Prefetch file using parallel processing toolbox in Matlab

Question

I'm new to parallel processing in Matlab. I'd like adjust an existing loop with code:

result=[];
load('a.mat','myvar');

for ind=1:100
  result(ind)=myvar; %some computationally 'heavy' function
end

While this loop is computing, I would like to pre-fetch the next file called 'b.mat'. When the loop finishes with the first file, it should check whether the next file is available, and if yes, continue on with the computation.

How can this be achieved? I looked into several tutorials but couldn't find an easy way to achieve this, but was presuming it should be fairly straightforward

Are all your files named with a sequential character like a, b, c, d, ...? — Tommaso Belluzzo
@TommasoBelluzzo: not really, all the paths are contained in a cell-array and the there's another loop cycling through the filenames. I didn't include this for simplicity. — user2305193
@CrisLuengo: Unfortunately yes, datafiles that are several GB, transferred to ram, and stored again on disk after running some computations — user2305193
Parallel processing doesn't seem to be a good idea to me. A parfor is enough to achieve it... but parallelizing huge files risks to saturate the available memory very quickly. — Tommaso Belluzzo

Edric Edric · Accepted Answer · 2018-02-06T09:47:50

Rather trying to pre-fetch, it might be simpler simply to wrap parfor around your whole loop. That way, each worker loads and then processes a file full of data. E.g.

fileNames = ...; % Get a list of files
parfor idx = 1:numel(fileNames)
    data = load(fileNames{idx}, 'myvar');
    result{idx} = doStuff(data.myvar);
end

Prefetch file using parallel processing toolbox in Matlab

1 Answers