Python Pillow opening 1-bit depth files with 8 bits

Question

I've got about 7GB of single-bit depth images (black and white BMP files), around 59 gigapixels total, which I need to paste together into a single square image. My system has 64GB RAM, so I figured this shouldn't be a problem to load all images at once, then create a new output image (should be the same size/resource-load), and paste the sub-images into it. Finally saving it to disk.

Well, watching top, I see after my images are loaded, the program is consuming 56GB RAM (i.e. 7GB * 8). During the pasting operation, the program gets killed.

Can I somehow change the Pillow open call to open in the native depth? Is there another image processing library someone might recommend instead (other languages are acceptable if that's the only obvious route...).

Here's the code:

import os
import math
import numpy as np
from rectpack import newPacker, PackingMode
import rectpack.packer
from PIL import Image, PILLOW_VERSION
dirname = 'output_12_15'
files = sorted(os.listdir(dirname))
images = []
num_pixels = 0
for i, file in enumerate(files[1:]):
    filepath = os.path.join(dirname, file)
    img = Image.open(filepath)
    images.append(img.copy())
    img.close()
    width, height = images[-1].size
    num_pixels += width*height
    if not i%1000:
        print(i)
print('number of total pixels: {}'.format(num_pixels))
edge_len = math.ceil(num_pixels**(1/2))
theorhetical_num_images_wide = math.ceil(edge_len/1366.)
target_width = (theorhetical_num_images_wide+1)*1366
target_height = target_width

packer = newPacker(mode=PackingMode.Offline, sort_algo=rectpack.packer.SORT_NONE, rotation=False)
# Add the rectangles to packing queue
for img in images:
    packer.add_rect(*img.size)
packer.add_bin(target_width, target_height)
print('starting to pack')
packer.pack()
print('finished packing, starting to paste image together')

if len(packer[0])!=len(images):
    print('packer failed, bin was too small!')
    import pdb;pdb.set_trace()
else:
else:
    max_y = 0
    max_x = 0
    blank = Image.new("1", (target_width,target_height))
    for i, rect in enumerate(packer[0]):
        x = rect.x
        y = rect.y
        w = rect.width
        h = rect.height
        if y+h>max_y:
            max_y = y+h
        if x+w>max_x:
            max_x = x+w
        blank.paste(images[i], (x,y))
    print('max_x {} max_y {} target_h {}'.format(max_x, max_y, target_height))
    blank = blank.convert('1', dither=Image.NONE)
    blank.save("59MP.bmp", "BMP")

UPDATE 1: I moved the code to another system with 256GB RAM, and found a bug after things got a little further (into the last FOR loop). But then it crashes again with this traceback:

0
1000
2000
3000
4000
5000
6000
number of total pixels: 59926682272
starting to pack
finished packing, starting to paste image together
max_x 247246 max_y 247243 target_h 247246
Traceback (most recent call last):
  File "stitch_bmps.py", line 75, in <module>
    blank.save("59MP.bmp", "BMP")
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/Image.py", line 2084, in save
    save_handler(self, fp, filename)
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/BmpImagePlugin.py", line 332, in _save
    + o32(offset)  # reserved
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/_binary.py", line 91, in o32le
    return pack("<I", i)
struct.error: 'I' format requires 0 <= number <= 4294967295

UPDATE 2: added a pdb.set_trace() call in the BmpImagePlugin file just before the error:

(Pdb) image
7641879368
(Pdb) offset+image
7641879430
(Pdb) (offset+image)<4294967295
False
(Pdb) (image)<4294967295
False

Sooo, I guess this is saying that BMP format, or the implementation PIL has for BMP, doesn't support such large images?

If you open all the input images and the output image at the same time, you will double the memory pressure. Why not create the output image and load a single input image at a time and paste it into the correct position before closing it and opening the next? Or do the job in two halves? — Mark Setchell
It highly likely PIL converts 1-bit per pixel image to 8-bit internally to make per-pixel operations much faster. The alternative would require a tremendous amount of masking and bit-twiddling to accomplish in a reasonable amount of time. That will likely be the case for most if not all other packages as well. — martineau
moved the code to a new system with 4x the RAM, now I've got an Exception — nmz787
added PDB trace info, showing the image variable in PIL is larger than 32-bits (2**32) — nmz787
The size of the BMP is given in a 32-bit integer in the BMP header at offset 2 bytes, so it cannot exceed 32-bits... en.wikipedia.org/wiki/BMP_file_format Maybe try saving as a PNG file instead. — Mark Setchell

mkrieger1 mkrieger1 · Accepted Answer · 2019-12-16T09:38:37

According to https://pillow.readthedocs.io/en/5.1.x/handbook/concepts.html, there is only an image mode

1 (1-bit pixels, black and white, stored with one pixel per byte)

but not one with 8 pixels per byte.

So it seems this is not possible with Pillow.

Python Pillow opening 1-bit depth files with 8 bits

1 Answers