Writing a 2D Numpy Array in a HDF5 Dataset

Question

So I want to write a 2D Numpy Array to a HDF5 file using Python (H5Py), however I am incapable of getting it work correctly. Here is what the dataset should look like

The Properties

The Data

Here is the code

    elements = {
        'Ti': ['47Ti', '49Ti'],
        'Cr': ['52Cr', '53Cr'],
        'Fe': ['54Fe', '57Fe'],
        'Mn': ['55Mn']}

    # arg3: signalData
    element_data = hdf5processor.process_signal_data(argv[3], elements)

    #hdf5processor.plot_elements(element_data)

    # arg4: outputFile
    hdf5processor.save_dataset(argv[4], elements, element_data)

    def save_dataset(filename, elements_list, element_data):
        hf = h5py.File(filename, 'a')

        elements_list_ascii = [n.encode("ascii", "ignore") for n in list(elements_list.keys())]

        elements_list_dataset = hf.create_dataset("spWork/ElementList", (len(elements_list_ascii), 1), data=elements_list_ascii, dtype=h5py.string_dtype())

        iostopes_used = np.array([['Element', 'Isotope(s)', 'Null', 'Null', 'Null'], ['Ti', '47Ti', '49Ti', 'Null', 'Null']])

        iostopes_used_dataset = hf.create_dataset("spWork/IsotopesUsed", (2, 5), data=iostopes_used, dtype=h5py.string_dtype())

        hf.close()

I'm trying to save the iostopes_used (2D Numpy String Array) to the HDF5 file as a variable length string like in the first and second image.

hpaulj hpaulj · Accepted Answer · 2020-04-18T00:09:10

http://docs.h5py.org/en/stable/strings.html is the relevant h5py docs page.

In [652]: iostopes_used = np.array([['Element', 'Isotope(s)', 'Null', 'Null', 'Null'], ['Ti', '47Ti', '
     ...: 49Ti', 'Null', 'Null']]) 
     ...:                                                                                              
In [653]: iostopes_used                                                                                
Out[653]: 
array([['Element', 'Isotope(s)', 'Null', 'Null', 'Null'],
       ['Ti', '47Ti', '49Ti', 'Null', 'Null']], dtype='<U10')
In [654]: f = h5py.File('names.h5','w')                                                                
In [655]: iostopes_used_dataset = f.create_dataset("data", data=iostopes_used)                         
---------------------------------------------------------------------------
....
TypeError: No conversion path for dtype: dtype('<U10')

If instead we convert the Py3 unicode strings to Py2 byte strings, the save works:

In [656]: iostopes_used_dataset = f.create_dataset("data", data=iostopes_used.astype('S10'))           
In [657]: iostopes_used_dataset[:]                                                                     
Out[657]: 
array([[b'Element', b'Isotope(s)', b'Null', b'Null', b'Null'],
       [b'Ti', b'47Ti', b'49Ti', b'Null', b'Null']], dtype='|S10')
In [658]: f.close()

===

Another route is variable length string objects, as illustrated in this recent SO question: Failing to write in hdf5 file

In [663]: dt = h5py.special_dtype(vlen=str)
In [665]: f.create_dataset('other', iostopes_used.shape, dtype=dt)                                     
Out[665]: <HDF5 dataset "other": shape (2, 5), type "|O">
In [666]: _[:] = iostopes_used                                                                         
In [667]: _[:]                                                                                         
Out[667]: 
array([['Element', 'Isotope(s)', 'Null', 'Null', 'Null'],
       ['Ti', '47Ti', '49Ti', 'Null', 'Null']], dtype=object)

Writing a 2D Numpy Array in a HDF5 Dataset

1 Answers