Python outputs strange bytearray

1
votes

Python sometimes generates strange cryptic byte arrays. I have no clue how to interpret them.

Consider the following example.

import struct
floats = [3.14, 2.7, 0.0, -1.0, 1.1]
s = struct.pack('f'*len(floats), *floats)
print("The bytes:")
print(s)

The struct.pack function should output the 'bytes-representation' of each value in the list. The list consists of 64-bit floating point numbers (my computer is 64-bit), so I would expect every float to be represented by 8 bytes:

 3.14 -> 0x40 0x09 0x1E 0xB8 0x51 0xEB 0x85 0x1F
 2.7  -> 0x40 0x05 0x99 0x99 0x99 0x99 0x99 0x9A
 0.0  -> 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
-1.0  -> 0xBF 0xF0 0x00 0x00 0x00 0x00 0x00 0x00
 1.1  -> 0x3F 0xF1 0x99 0x99 0x99 0x99 0x99 0x9A

By the way, I have used the following website to make the proper conversion: http://babbage.cs.qc.cuny.edu/IEEE-754.old/Decimal.html

Unfortunately, Python does not output those bytes I would expect. Instead, Python outputs some very cryptic list of bytes. But is this really a list of bytes? The thing that Python outputs is so strange:

b'\xc3\xf5H@\xcd\xcc,@\x00\x00\x00\x00\x00\x00\x80\xbf\xcd\xcc\x8c?'

Please help me to understand what Python is doing here.

EDIT

Apparently I should use 'd' instead of 'f', since I'm using double precision floating point numbers on my machine. Thank you mr. Rad Lexus for your answer. But I'm still a bit puzzled about the Python outputs. Let me clarify.

I start with the following code that you gave me:

import struct
floats = [3.14, 2.7, 0.0, -1.0, 1.1]
s = []
for f in floats:
    s.append(struct.pack('d', f))

Before proceeding, I inspect the object s to get a grasp of what is happening. This is what I get from s:

>>> s
[ b'\x1f\x85\xebQ\xb8\x1e\t@', 
  b'\x9a\x99\x99\x99\x99\x99\x05@',
  b'\x00\x00\x00\x00\x00\x00\x00\x00', 
  b'\x00\x00\x00\x00\x00\x00\xf0\xbf',
  b'\x9a\x99\x99\x99\x99\x99\xf1?'     ]

Some entries in s are 8 bytes long. That is what I would expect. But some entries are shorter. Anyway, there is no entry that gives the correct 8-byte representation of the corresponding float number - except for the float 0.0.

Your code continues with some magic to extract the actual correct 8 bytes per float:

print("The bytes:")
for floatInHex in s:
    for byteval in floatInHex:
        print ('%02x' % byteval, end="")

Now we get the correct result. But why is the s object not yet containing the correct 8-bytes per float, to begin with? Why is this extra magic needed?

3
pythonpython-3.xdoublebytearray

3 Answers

2
votes
>>> binascii.hexlify(struct.pack('>d', floats[0]))
b'40091eb851eb851f'

>>> import struct
>>> import binascii
>>>
>>> floats = [3.14, 2.7, 0.0, -1.0, 1.1]
>>> s = struct.pack('>' + 'd'*len(floats), *floats)
>>> binascii.hexlify(s)
b'40091eb851eb851f400599999999999a0000000000000000bff00000000000003ff199999999999a'

If you want to get each float representation separately, you need to iterate them and conver them. (using loop or list comprehension, ..)

>>> for f in floats:
...     print(' '.join('0x{:02x}'.format(c) for c in struct.pack('>d', f)))
...
0x40 0x09 0x1e 0xb8 0x51 0xeb 0x85 0x1f
0x40 0x05 0x99 0x99 0x99 0x99 0x99 0x9a
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0xbf 0xf0 0x00 0x00 0x00 0x00 0x00 0x00
0x3f 0xf1 0x99 0x99 0x99 0x99 0x99 0x9a
1
votes

You want to unpack as double but in the program you use the specifier for float: f. So that's what you get:

c3 f5 48 40 for 3.14

(See python struct pack double for why you see some ASCII characters.)

This code will print out one line of hex per number:

import struct
floats = [3.14, 2.7, 0.0, -1.0, 1.1]
s = []
for f in floats:
    s.append(struct.pack('d', f))
print("The bytes:")
for floatInHex in s:
  for byteval in floatInHex:
    print ('%02x' % byteval, end=""),
  print ()

Result:

The bytes:
1f85eb51b81e0940
9a99999999990540
0000000000000000
000000000000f0bf
9a9999999999f13f
0
votes

Python is outputting that "strange" behavior on print because the print function will try to print any ASCII-printable character as ASCII, not hex.

It prints correctly with binascii because that never prints ASCII characters, only hex characters.

Had me fooled too! Was about to ask the same question, then found this here: Python Bytearray Printing