\[Errno -101\] NetCDF: HDF error when opening netcdf file

5
votes

I have this error when opening my netcdf file. The code was working before. How do I fix this ?

Traceback (most recent call last):

File "", line 1, in ...

File "file.py", line 71, in gather_vgt return xr.open_dataset(filename)

File "/.../lib/python3.6/site-packages/xarray/backends/api.py", line 286, in open_dataset autoclose=autoclose)

File "/.../lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 275, in open ds = opener()

File "/.../lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 199, in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, **kwargs)

File "netCDF4/_netCDF4.pyx", line 2015, in netCDF4._netCDF4.Dataset.init

File "netCDF4/_netCDF4.pyx", line 1636, in netCDF4._netCDF4._ensure_nc_success

OSError: [Errno -101] NetCDF: HDF error: b'file.nc'

When I try to open the same netcdf file with h5py I get this error :

OSError: Unable to open file (file locking disabled on this file system (use HDF5_USE_FILE_LOCKING environment variable to override), errno = 38, error message = '...')

2
hdf5netcdfh5pynetcdf4

2 Answers

9
votes

You must be in this situation :

  • your HDF5 library has been updated (1.10.1) (netcdf uses HDF5 under the hood)

  • your file system does not support the file locking that the HDF5 library uses.

In order to read your hdf5 or netcdf files, you need set this environment variable : 

 HDF5_USE_FILE_LOCKING=FALSE

For references, this was introduced in HDF5 version 1.10.1,

Added a mechanism for disabling the SWMR file locking scheme.

The file locking calls used in HDF5 1.10.0 (including patch1)
will fail when the underlying file system does not support file
locking or where locks have been disabled. To disable all file
locking operations, an environment variable named
HDF5_USE_FILE_LOCKING can be set to the five-character string
'FALSE'. This does not fundamentally change HDF5 library
operation (aside from initial file open/create, SWMR is lock-free),
but users will have to be more careful about opening files
to avoid problematic access patterns (i.e.: multiple writers) >that the file locking was designed to prevent.

Additionally, the error message that is emitted when file lock
operations set errno to ENOSYS (typical when file locking has been
disabled) has been updated to describe the problem and potential
resolution better.

(DER, 2016/10/26, HDFFV-9918)

2
votes

In my case, the solution as suggested by @Florian did not work. I found another solution, which suggested that the order in which h5py and netCDF4 are imported matters (see here).

And, indeed, the following works for me:

from netCDF4 import Dataset
import h5py

Switching the order results in the error as described by OP.