Accessing HDF5 attributes efficiently using h5py

Question

I'm reading attribute data for about 10-15 groups in a HDF5 file using h5py and then adding the data to a python dictionary to describe the file structure, which I use later to analyse and access the rest of the datasets when required. Most attribute sets consist of about 10 keys, off which I'm usually only interested in about 4-8 of.

struct = {}
temp = h5py.File(fname, 'r')
struct['a'] = dict(temp['Timestep']['0.1']['Groups']['1'].attrs.items())
struct['b'] = dict(temp['Timestep']['0.1']['Groups']['2'].attrs.items())
struct['c'] = dict(temp['Timestep']['0.1']['Groups']['3'].attrs.items())
...
...
tempt.close()

I do this for each .h5 file since each file represents a new time-step of simulation data. With each simulation usually consisting of a few thousand time-steps, I seem to be wasting a lot of time reading all the necessary meta-data required for later use.

I've done some simple checks on performance to see if there's anything I could do to make things quicker like accessing only the keys I want or converting to lists instead of dictionaries, but neither of these are significantly quicker.

%timeit a = temp['Timestep']['0.1']['Groups']['1'].attrs.items()
134 µs ± 3.74 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit a = dict(temp['Timestep']['0.1']['Groups']['1'].attrs.items())
809 µs ± 32.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit 
c = {}
for key in temp['Timestep']['0.1']['Groups']['1'].attrs.keys():
    c[key] = (temp['Timestep']['0.1']['Groups']['1'].attrs[key])
1.99 ms ± 85.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit 
   ...: b1 = temp['Timestep']['0.1']['Groups']['1'].attrs['var1']
   ...: b2 = temp['Timestep']['0.1']['Groups']['1'].attrs['var2']
   ...: b3 = temp['Timestep']['0.1']['Groups']['1'].attrs['var5']
   ...: b4 = temp['Timestep']['0.1']['Groups']['1'].attrs['var6']
   ...: b5 = temp['Timestep']['0.1']['Groups']['1'].attrs['var9']
   ...: b6 = temp['Timestep']['0.1']['Groups']['1'].attrs['var11']
1.44 ms ± 52.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit d = list(temp['Timestep']['0.1']['Groups']['1'].attrs.items())
830 µs ± 29.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

From the results it seems that accessing the data (which creates a ItemsViewHDF5 object) is relatively quick at about 100us, but converting it to a dictionary or list takes a lots more time - about 800us. Reading the whole set of attributes is much quicker than just reading the ones of interest individually.

Does anyone know of a way to get the attribute data into some other data-structure that is available once the h5file has been read and closed? I'd like to be able to do this closer to the 100us rather than the 1ms mark!

Clarification: For clarity, the attribute data is of mixed types. For example var1 is typically a string of the user assigned name, and then var2 would typically be an int that is the size (number of rows in the dataset). Other variables tend to be vectors (x,y,z) like velocity that are stored as arrays, but sometimes single floats.

It's hard to comment on data structures without knowing what kind of data you're saving, and how it varies from group to group. That said, I have a suggestion: Have you considered a NumPy Record Array (recarray)? It is a 2d table that can store mixed data types (floats, ints, strings, etc). Recarray fields (the columns) are labeled, so your attribute names map to the field names, and each group's attributes become a row in the recarray. This assumes you are saving common attributes from each group. — kcw78
BTW, you can simplify your code with one of the visitor methods (.visit() or .visititems()) on your file object: temp.visititems(-callable function-). It will recursively visit each object in this group and subgroups. You supply the callable function (which would load your attributes into a data structure). — kcw78
@kcw78 Added info on the data types. I have tried converting directly to a numpy recarray by calling np.rec.array(temp[...][...][...][...].attrs.items()) but that leads to a value error. I think that recarrays need to have a lot of information defined at creation time. Simply casting to a dictionary is appealing from a simplicity point of view. Also, I actually started off using .visit() to walk through the file tree but found out it was slower than explicitly calling everything. Maybe I didn't do a very good job of writing my function, but I think it was about 5-10x slower. — jpmorr
ItemsViewHDF5 object doesn't actually load data from the file. Like adict.items() it's action waiting to be done. list(...attrs.items()) or dict does the actual loading. That's why it takes longer. But without a sample file, it will be hard to replicate or improve those observations. While I've answered many h5py questions, I haven't done much with attrs, and probably don't have a good dummy file to test them on. — hpaulj
recarrays aren't complicated, but generally you need to define the dtype. There are several ways. You can use a dictionary with 2 lists (names and formats) like this: attr_dtype = {'names':['attr_1','attr_2’,...,'attr_N’], 'formats':['S20', int, ... ,float] }, After creating, reference when you create the recarray. Then you can add attributes using NumPy notation: attr_recarr['attr_1'][row] = attr_1_value. If you don't know your data types in advance, you can create the dtype programmatically by constructing the name and format lists from your data. — kcw78

hpaulj hpaulj · Accepted Answer · 2020-08-03T20:11:36

Make a simple h5 with a group and adds some attrs to that:

In [2]: f = h5py.File('testattr.h5','w')                                                             
In [3]: g = f.create_group('test')                                                                   
In [4]: g.attrs                                                                                      
Out[4]: <Attributes of HDF5 object at 139650561822480>
In [5]: g.attrs['var1'] = 'foobar'                                                                   
In [6]: g.attrs['var2'] = 23                                                                         
In [7]: g.attrs['var3'] = np.arange(24).reshape(2,3,4)                                               
In [8]: g.attrs['var4'] = np.pi

Get the itemsview, and then load with list (or dict):

In [9]: g.attrs.items()                                                                              
Out[9]: ItemsViewHDF5(<Attributes of HDF5 object at 139650561822480>)
In [10]: list(g.attrs.items())                                                                       
Out[10]: 
[('var1', 'foobar'),
 ('var2', 23),
 ('var3', array([[[ 0,  1,  2,  3],
          [ 4,  5,  6,  7],
          [ 8,  9, 10, 11]],
  
         [[12, 13, 14, 15],
          [16, 17, 18, 19],
          [20, 21, 22, 23]]])),
 ('var4', 3.141592653589793)]

Getting viewer is trivial timewise:, but actually loading the data to list or dict takes time - actual format doesn't matter:

In [11]: timeit g.attrs.items()                                                                      
4.7 µs ± 177 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [13]: timeit list(g.attrs.items())                                                                
273 µs ± 2.71 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [15]: timeit dict(g.attrs.items())                                                                
272 µs ± 287 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Adding another attribute (relatively large), ups the load time a bit:

In [26]: g.attrs['var5'] = np.random.rand(1000)  
In [29]: timeit list(g.attrs.values())                                                               
337 µs ± 633 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Compare that with using a dataset:

In [30]: g.create_dataset('oned', data=np.random.rand(1000))                                         
Out[30]: <HDF5 dataset "oned": shape (1000,), type "<f8">
In [31]: f.flush()                                                                                   
In [32]: g['oned']                                                                                   
Out[32]: <HDF5 dataset "oned": shape (1000,), type "<f8">
In [33]: g['oned'][:];                                                                               
In [34]: timeit g['oned']                                                                            
64.4 µs ± 143 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [35]: timeit g['oned'][:]                                                                         
239 µs ± 188 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [36]: ds = g['oned']                                                                              
In [37]: timeit ds[:]                                                                                
151 µs ± 2.73 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Accessing HDF5 attributes efficiently using h5py

1 Answers