Is there any parallel way of accessing Netcdf files in Python

5
votes

Is there any way of doing parallel IO for Netcdf files in Python?

I understand that there is a project called PyPNetCDF, but apparently it's old, not updated and doesn't seem to work at all. Has anyone had any success with parallel IO with NetCDF in Python at all?

Any help is greatly appreciated

2
pythonioparallel-processingnetcdf

2 Answers

2
votes

It's too bad PyPnetcdf is not a bit more mature. I see hard-coded paths and abandoned domain names. It doesn't look like it will take a lot to get something compiled, but then there's the issue of getting it to actually work...

  • in setup.py you should change the library_dirs_list and include_dirs_list to point to the places on your system where Northwestern/Argonne Parallel-NetCDF is installed and where your MPI distribution is installed.

  • then one will have to go through and update the way pypnetcdf calls pnetcdf. A few years back (quite a few, actually) we promoted a lot of types to larger versions.

1
votes

I haven't seen good examples from either of the two python NetCDF modules, see https://github.com/Unidata/netcdf4-python/issues/345 However, if You only need to read files and they are NetCDF4 format, You should be able to use HDF5 directly -- http://docs.h5py.org/en/latest/mpi.html because NetCDF4 is basically HDF5 with restricted data model. Probably won't work with NetCDF3.