Reading a pre-formatted binary file in Python

Question

So, I created a simple binary file using MATLAB that has the following structure:

file.test
--------
[record_type] = 1 % 'int', 4 bytes, record_type = 1 means a string is read next
[string_length] = len(str) % 'int', 4 bytes, tells us how many bytes to read
[string] = '...' % 'char', the number of bytes in string length

[record_type] = 2 % 'int', 4 bytes, record_type = 2 means a vector will be read
[rows] = size(vector,1) % 'int', 4 bytes
[columns] = size(vector,2) % 'int', 4 bytes
[vector] = (vector) % 'double'

I can read this file back in MATLAB with the following code (this is just the read portions of the code, I do have error checking and other things):

fid=fopen('file.test','rb')
record_names={}
record_data=[]

while ~feof(fid)
    [record_name_type,count]=fread(fid,1,'int');
    if count == 0 % reached eof
        break;
    end
    record_name_length=fread(fid,1,'int');
    record_names{end+1}=char(fread(fid,record_name_length,'char')');
    % read vector
    record_type=fread(fid,1,'int');
    rows=fread(fid,1,'int');
    cols=fread(fid,1,'int');
    record_data{end+1}=fread(fid,[rows,cols],'double');
end

So, I had to translate this same function into Python 2.7. Unfortunately, it's proving to be very difficult. I can't just call fread anymore and tell it how many elements or bytes I want to read. I have tried the following code with structs:

def read_file(filename):
    # checking to see if file exists / other checks happen
    fid=open(filename,'rb')
    record_data=[]
    record_names=[]
    result=read_record(fid)
    while result:
        record_names.append(result['record_name'])
        record_names.append(result['data'])
        result=read_record(fid)
    fid.close()
    return (record_names, record_data)

def read_record(fid):
    try:
        # read name
        record_name_type = struct.unpack('i',fid.read(struct.calcsize('i')))[0]
        record_name_len = struct.unpack('i',fid.read(struct.calcsize('i')))[0]
        record_name = struct.unpack('s',fid.read(record_name_len))[0]
        # read vector
        record_type = struct.unpack('i',fid.read(struct.calcsize('i')))[0]
        rows = struct.unpack('i',fid.read(struct.calcsize('i')))[0]
        cols = struct.unpack('i',fid.read(struct.calcsize('i')))[0]
        record_data = numpy.array(struct.unpack('%dd' % rows,fid.read(rows*struct.calcsize('d'))),dtype=float)
        # store into result and return
        result=OrderedDict()
        result['record_name']=record_name
        result['data']=record_data
    except struct.error as e:
        print e
        result=None
    return result

Now when running my method in python, I get the following error:

unpack requires a string argument of length 1 which is coming from the except strcut.error as e portion. I had a feeling I was reading the vector incorrectly but I don't know how I'm reading the string incorrectly.

Does anyone know of an easier way to read this binary file? Or is there some tutorial I could follow to help me understand how to properly use structs in Python? I'm really new to it and especially this area in Python.

The statement: record_name = struct.unpack('s',fid.read(record_name_len))[0] caused the exception. You should specify the length of string in the format paremeter: '%ds' % record_name_len. — acw1668
@acw1668 But then how do I read it? Doing record_name = struct.unpack('%ds' % record_name_len,fid.read(record_name_len))[0] returns a TypeError: unpack expected 2 arguments, got 3. If I put only record_name_len in there, does it just know to use read()? I'm kind of confused D: — Alex

fjarri fjarri · Accepted Answer · 2016-08-19T01:18:03

You can use Construct:

from construct import *

MyData = Struct(
    "MyData",
    SBInt32("record_type"),
    Switch("value", lambda ctx: ctx.record_type,
    {
        1: Embed(Struct(
            'string_data',
            SBInt32("string_length"), Bytes("string", lambda ctx: ctx.string_length))),
        2: Embed(Struct(
            'vector_data',
            SBInt32("rows"), SBInt32("columns"),
            Array(lambda ctx: ctx.rows * ctx.columns, LFloat64("vector"))))
    }
    )
)

s = MyData.build(Container(record_type=1, string_length=4, string='abcd'))
print(repr(s))
print(MyData.parse(s))

s = MyData.build(Container(record_type=2, rows=2, columns=3, vector=[1, 2, 3, 4, 5, 6]))
print(repr(s))
print(MyData.parse(s))

The output:

'\x00\x00\x00\x01\x00\x00\x00\x04abcd'
Container:
    record_type = 1
    string_length = 4
    string = 'abcd'
'\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x08@\x00\x00\x00\x00\x00\x00\x10@\x00\x00\x00\x00\x00\x00\x14@\x00\x00\x00\x00\x00\x00\x18@'
Container:
    record_type = 2
    rows = 2
    columns = 3
    vector = [
        1.0
        2.0
        3.0
        4.0
        5.0
        6.0
    ]

Reading a pre-formatted binary file in Python

1 Answers