Python Converts integer into a bit number of specific length, fast

Question

I am trying to delta compress a list of pixels and store them in a binary file. I have managed to do this however the method I found takes ~4 minutes a frame.

def getByte_List(self):
    values = BitArray("")
    for I in range(len(self.delta_values)):
        temp = Bits(int= self.delta_values[I], length=self.num_bits_pixel)
        values.append(temp)
    ##start_time = time.time()
    bit_stream = pack("uint:16, uint:5, bits", self.intial_value, self.num_bits_pixel, values)
    ##end_time = time.time()
    ##print(end_time - start_time)

    # Make sure that the list of bits contains a multiple of 8 values
    if (len(bit_stream) % 8):
        bit_stream.append(Bits(uint=0, length = (8-(len(bit_stream) % 8))))                           #####Append? On a pack? (Only work on bitarray? bit_stream = BitArray("")

    # Create a list of unsigned integer values to represent each byte in the stream
    fmt = (len(bit_stream)/8) * ["uint:8"]
    return bit_stream.unpack(fmt)

This is my code. I take the initial value, the number of bits per pixel and the delta values and convert them into bits. I then byte align and take the integer representation of the bytes and use it elsewhere. The problem areas are where I convert each delta value to bits(3min) and where I pack(1min). Is it possible to do what I am doing faster or another way to pack them straight into integers representing bytes.

For this kind of manipulation, you most certainly want to use a lower-level language, or the excellent numpy. Straight python is not really suitable for this — loopbackbee

kindall kindall · Accepted Answer · 2013-10-16T15:13:39

From a quick Google of the classes you're instantiating, it looks like you're using the bitstring module. This is written in pure Python, so it's not a big surprise that it's pretty slow. You might look at one or more of the following:

struct - a module that comes with Python that will let you pack and unpack C structures into constituent values
bytearray - a built-in type that lets you accumulate, well, an array of bytes, and has both list-like and string-like operations
bin(x), int(x, 2) - conversion of numbers to a binary representation as a string, and back - string manipulations can sometimes be a reasonably efficient way to do this
bitarray - native (C) module for bit manipulation, looks like it has similar functionality to bitstring but should be much faster. Available here in form suitable for compiling on Linux or here pre-compiled for Windows.
numpy - fast manipulation of arrays of various types including single bytes. Kind of the go-to module for this sort of thing, frankly. http://www.numpy.org/

Python Converts integer into a bit number of specific length, fast

1 Answers