python - HDF5 file created with h5py can't be opened by h5py

Question

I created an HDF5 file apparently without any problems, under Ubuntu 12.04 (32bit version), using Anaconda as Python distribution and writing in ipython notebooks. The underlying data are all numpy arrays. For example,

import numpy as np
import h5py

f = h5py.File('myfile.hdf5','w')

group = f.create_group('a_group')

group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')

If I try to open this file from a new iypthon notebook, though, I get an error message:

f = h5py.File('myfile.hdf5', "r")

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-4-b64ac5089cd4> in <module>()
----> 1 f = h5py.File(file_name, "r")

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, driver, libver, userblock_size, **kwds)
    220 
    221             fapl = make_fapl(driver, libver, **kwds)
--> 222             fid = make_fid(name, mode, userblock_size, fapl)
    223 
    224         Group.__init__(self, fid)

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in make_fid(name, mode, userblock_size, fapl, fcpl)
     77 
     78     if mode == 'r':
---> 79         fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl)
     80     elif mode == 'r+':
     81         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

/home/sarah/anaconda/lib/python2.7/site-packages/h5py/h5f.so in h5py.h5f.open (h5py/h5f.c:1741)()

IOError: Unable to open file (Unable to find a valid file signature)

Can you tell me what that missing file signature is? Did I miss something when I created the file?

Did you f.close() your writable file before trying to open it again? Also, your example code is not executable: the variables Mfrgroup, fgroup_ID, pos, Msgroup sgroup_ID and names are not defined. — farenorth
It is often recommended when writing to or from files (using any file constructor, not just h5py) to use the with statement. For example with h5py.File('myfile.hdf5', 'w') as f:. This makes it so that you don't have to explicitly close the file. On the other hand, it makes it difficult to debug file I/O interactively. — farenorth
Also, do you really need to create all of those variables in your dataset to generate this error? Do you get the same error if you create only one of those variables? — farenorth
You were right, I did forget to f.close() the file! I normally use the with statement, but this time followed a tutorial and of course forgot that... Do you want to write an answer for that, or should I? Or is there another way to mark this question as solved? — Lilith-Elina
Great! I don't know much about iPython notebooks, but based on this error I am guessing that they use the same interpreter (kernel/session). This page has a discussion of separating the kernel from the notebook, which is probably apropos to your issue; i.e. perhaps you are using iPy notebooks prior to this change? Those types of things could be documented in the answer you provide. — farenorth

Lilith-Elina Lilith-Elina · Accepted Answer · 2014-11-04T08:42:55

Since we resolved the issue in the comments on my question, I'm writing the results out here to mark it as solved.

The main problem was that I forgot to close the file after I created it. There would have been two simple options, either:

import numpy as np
import h5py

f = h5py.File('myfile.hdf5','w')
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
f.close()

or, my favourite because the file is closed automatically:

import numpy as np
import h5py

with h5py.File('myfile.hdf5','w') as f:
    group = f.create_group('a_group')
    group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')

python - HDF5 file created with h5py can't be opened by h5py

2 Answers