How to set parameters in Python zlib module

Question

I want to write a Python program that makes PNG files. My big problem is with generating the CRC and the data in the IDAT chunk. Python 2.6.4 does have a zlib module, but there are extra settings needed. The PNG specification REQUIRES the IDAT data to be compressed with zlib's deflate method with a window size of 32768 bytes, but I can't find how to set those parameters in the Python zlib module.

As for the CRC for each chunk, the zlib module documentation indicates that it contains a CRC function. I believe that calling that CRC function as crc32(data,-1) will generate the CRC that I need, though if necessary I can translate the C code given in the PNG specification.

Note that I can generate the rest of the PNG file and the data that is to be compressed for the IDAT chunk, I just don't know how to properly compress the image data for the IDAT chunk after implementing the initial filtering step.

EDITED:

The problem with PyPNG is that it will not write tEXt chunks. A minor annoyance is that one has to manipulate the image as (R, G, B) data; I'd prefer to manipulate palette values of the pixels directly and then define the associations between palette values and color data. I'm also left unsure if PyPNG takes advantage of the "compression" allowed by using 1-, 2-, and 4- bit palette values in the image data to fit more than one pixel in a byte.

(I'm the maintainer of PyPNG) Palette based images aren't something I play with much, but PyPNG does have pretty much full support for them. Yes, you can do palette images just like you want to, see this example from the documentation (I agree this might not be clear). Yes you can have 2-bits per pixel palette images and this will pack 4 pixels into a byte. Simply specify bitdepth=2 when creating the png.Writer instance (this is not clear). And of course, that work for other bitdepths too. — David Jones
You're right that you can't add a tEXt chunk with PyPNG. That's Issue 4, and now I know someone wants it, I may well add the feature soon. Would you like to work together to create a good API? — David Jones

John Machin John Machin · Accepted Answer · 2011-01-06T22:13:58

Short answer: (1) "deflate" and "32Kb window" are the defaults (2) uses adler32 not crc32

Long answer:

""" The PNG specification REQUIRES the IDAT data to be compressed with zlib's deflate method with a window size of 32768 bytes, but I can't find how to set those parameters in the Python zlib module. """

You don't need to set them. Those are the defaults.

If you really want to specify non-default arguments to zlib, you can use zlib.compressobj() ... it has several args that are not documented in the Python docs. Reading material:

source: Python's gzip.py (see how it calls zlib.compressobj)

source: Python's zlibmodule.c (see its defaults)

SO: This question (see answers by MizardX and myself, and comments on each)

docs: The manual on the zlib site

"""As for the CRC for each chunk, the zlib module documentation indicates that it contains a CRC function. I believe that calling that CRC function as crc32(data,-1) will generate the CRC that I need, though if necessary I can translate the C code given in the PNG specification."""

Please check out the zlib specification aka RFC 1950 ... it says that the checksum used is adler32

The zlib compress or compressobj output will include the appropriate CRC; why do you think that you will need to do it yourself?

Edit So you do need a CRC-32. Good news: zlib.crc32() will do the job:

Code:

import zlib

crc_table = None

def make_crc_table():
  global crc_table
  crc_table = [0] * 256
  for n in xrange(256):
    c = n
    for k in xrange(8):
        if c & 1:
            c = 0xedb88320L ^ (c >> 1)
        else: 
            c = c >> 1
    crc_table[n] = c

make_crc_table()    

"""
/* Update a running CRC with the bytes buf[0..len-1]--the CRC
should be initialized to all 1's, and the transmitted value
is the 1's complement of the final running CRC (see the
crc() routine below)). */
"""
def update_crc(crc, buf):
  c = crc
  for byte in buf:
    c = crc_table[int((c ^ ord(byte)) & 0xff)] ^ (c >> 8)
  return c

# /* Return the CRC of the bytes buf[0..len-1]. */
def crc(buf):
  return update_crc(0xffffffffL, buf) ^ 0xffffffffL

if __name__ == "__main__":
    tests = [
        "",
        "\x00",
        "\x01",
        "Twas brillig and the slithy toves did gyre and gimble in the wabe",
        ]

    for test in tests:
        model = crc(test) & 0xFFFFFFFFL
        zlib_result = zlib.crc32(test) & 0xFFFFFFFFL
        print (model, zlib_result, model == zlib_result)

Output from Python 2.7 is below. Also tested with Python 2.1 to 2.6 inclusive and 1.5.2 JFTHOI.

(0L, 0L, True)
(3523407757L, 3523407757L, True)
(2768625435L, 2768625435L, True)
(4186783197L, 4186783197L, True)

How to set parameters in Python zlib module

6 Answers