I am working with large csv files and trying to convert them to hdf5 format. When I try to view the output using HDFviewer, I get this strange segmented output consisting multiple datasets instead of one nice cohesive dataset I see with other people's hdf5 files. When I try to open block2 values, it will crash the application I try to open it with. Attached is photo of hdfviewer opening a hdf5 file I generated with pandas.
dataset is split into several blocks and the header and index are on axis=1, and axis=0
Example of dataset with headers (col1, col2, col3) with type int, string(object), and float:
+------+------+-------+-----+
| Col1 | Col2 | Col3 | ... |
+------+------+-------+-----+
| 1 | 0x00 | 100.1 | ... |
| 2 | 0x00 | 100.3 | ... |
| 3 | 0x00 | 132.3 | ... |
+------+------+-------+-----+
Example of code csv->hdf5:
df = pd.DataFrame()
df = pd.read_csv(csv_file)
df.to_hdf('example.h5', '/data', complib='zlib', complevel=9)
What is going on here?
EDIT: to clarify, I can open my generated .h5 files fine in python using pd.read_hdf()...but applications such as JMP and hdfview display the image I have attached and cannot open it properly and even crash.
EDIT: Okay I know why Pandas splits my dataset into three datasets: because it's more efficient to group datasets of the same type so block0 is a dataset of type int, block1 type float, and block2 type string. However, when I try to open block2 it will freeze hdfviewer. And block2 won't even show up on JMP.
EDIT: OKAY I THINK I FOUND WHAT IS CAUSING THE STRING BLOCK TO BREAK, BUT I DO NOT KNOW HOW TO FIX THIS
Here is an example dataset:
+------+-------+-------+
| | Col1 | Col2 |
+------+-------+-------+
| 0 | hello | the |
| 2 | world | computer|
| 3 | lol | is |
+------+------+----------+
Here is the output when converting to hdf5:
(128, 4, 149, 185, 0, 0, 0, 0, 0, 0, 0, 140, 21, 110, 117, 109, 112, 121, 46, 99, 111, 114, 101, 46, 109, 117, 108, 116, 105, 97, 114, 114, 97, 121, 148, 140, 12, 95, 114, 101, 99, 111, 110, 115, 116, 114, 117, 99, 116, 148, 147, 148, 140, 5, 110, 117, 109, 112, 121, 148, 140, 7, 110, 100, 97, 114, 114, 97, 121, 148, 147, 148, 75, 0, 133, 148, 67, 1, 98, 148, 135, 148, 82, 148, 40, 75, 1, 75, 3, 75, 2, 134, 148, 104, 3, 140, 5, 100, 116, 121, 112, 101, 148, 147, 148, 140, 2, 79, 52, 148, 75, 0, 75, 1, 135, 148, 82, 148, 40, 75, 3, 140, 1, 124, 148, 78, 78, 78, 74, 255, 255, 255, 255, 74, 255, 255, 255, 255, 75, 63, 116, 148, 98, 136, 93, 148, 40, 140, 5, 104, 101, 108, 108, 111, 148, 140, 3, 116, 104, 101, 148, 140, 5, 119, 111, 114, 108, 100, 148, 140, 8, 99, 111, 109, 112, 117, 116, 101, 114, 148, 140, 3, 108, 111, 108, 148, 140, 2, 105, 115, 148, 101, 116, 148, 98, 46)
Now run an ascii-> text converter and it shows:
¹numpy.core.multiarray_reconstructnumpyndarrayKCbR(KKKhdtypeO4KKR(K|NNNJÿÿÿÿJÿÿÿÿK?tb](hellotheworldcomputerlolisetb
so it's writing actual python code to the hdf5 file and breaking everything. Is this a bug with pandas?
