Compress data to chunks using Zlib

Question

I want to compress a file with size > 50mb with zlib. But I want the compressed file to be in chunks (for example, 32Kb chunks).

I don't want to read 32Kb of uncompressed file and compress it. The compressed file should contain 32kb compressed chunks. Compression should be reset after avail_out = 32kb.

I need to do it only with zlib and this is for random access.

This is what I have so far:

do {
    strm.avail_in = fread(in, 1, CHUNK, source);

    flush = feof(source) ? Z_FINISH : Z_FULL_FLUSH;
    strm.next_in = in;
    strm.next_out = out;

    strm.avail_out = CHUNK;
    deflate(&strm, flush);

    if (strm.avail_out == 0)
        have = tmp;
    else
        have = CHUNK - strm.avail_out;

    fwrite(out, 1, have, dest);

    tmp = strm.avail_out;

    if (tmp == 0)
        deflateReset(&strm);

} while (flush != Z_FINISH);

but there are several problems in this code.

When I reset the compression every 32Kb, I lose some compressed bytes that haven't been written yet because avail_out is full; and if I try to write them then the compressed chunk wouldn't be 32Kb anymore.
If I knew how much data has been compressed when avail_out is filled, I could reset the file position and begin compression from there.

What have you tried? Can you add some code of your current solution? I'm asking because this looks like a uni assignment, where you want someone else to do the work for you. — Augusto

Mark Adler Mark Adler · Accepted Answer · 2015-06-01T05:29:06

It is not possible to assure that you can exactly fill the desired size, but you can get very close. See fitblk.c for how to do it.

Compress data to chunks using Zlib

1 Answers