Socket operation errors with MATLAB and the Parallel Computing Toolbox

Question

We're using the MATLAB Parallel Computing Toolbox for one of our projects and we intermittently get the following error:

The client lost connection to lab 2. This might be due to network problems, or the pmode parallel job might have errored. This is causing: java.io.IOException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

All of the machines involved are running Windows XP SP3 32-bit, using MATLAB 2009a. The error happens intermittently from different labs.

Any ideas would be much appreciated.

Here's a rough/generic snippet of code as to what it's doing:

R = findResource();
matlabpool('open', 'local', R.ClusterSize);
spmd
... some loadlibrary() calls ...
end

parfor i = 1:num
  ... big calculations ...
end
%clear
matlabpool close

How many workers are you running per machine? Are you transferring lots of data to/from the workers? Any chance you could post some code? — Edric
@Jonas - unfortunately I'm not sure how to do that since the Java components seem to be internal to Matlab. @Edric - we're using 8 workers. I'm not sure of the exact volume of the data, but we encounter this problem intermittently regardless of data set (some of our large data sets hit this, as do some of our relatively smaller data sets). Unfortunately it's proprietary code, but I'll put some of the more generic snippets in my question. Thanks! — russcollier
In recent versions of Matlab, there is a slider in preferences/general/java heap memory that you can adjust. Otherwise, or if you want to assign even more memory, look here: mathworks.com/support/solutions/en/data/1-18I2C — Jonas
Do either of the suggestions here help? blogs.msdn.com/b/sql_protocols/archive/2009/03/09/… — richardthe3rd

russcollier russcollier · Accepted Answer · 2013-02-11T14:54:31

The problem turned out to have nothing to do with Matlab at all.

An underlying external messaging library we were using/loading (via those loadlibrary() calls) and then using the "big calculations" section was exhausting the nonpaged pool memory size on 32-bit versions of Windows XP and Windows Server 2003 by creating/using a large number of socket buffers inside the bowels of the messaging library. The problem basically went away by switching to a 64-bit OS (Windows Server 2008 R2).

http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx

Socket operation errors with MATLAB and the Parallel Computing Toolbox

1 Answers