Determine the compressed block size with zlib/minizip

Question

I'm trying to make a cross-platform Appx packager (for Windows store apps). The format is just a zip with some metadata files.

I need to make a XML file with hashes of the files as it is required by the format. The problem is that it needs one hash for each 64KB of uncompressed data of each file and also the compressed size of such 64KB block. How do I do that?

I'm using the Minizip library and can also use zlib directly if need be.

It seems to me that the I need to either avoid compressing the files (which I'd like to avoid) or skipping Minizip's compression and using raw stuff from zlib for deflating the files (I'm not familiar with zlib's API to know if it's feasible to make each 64KB block individually enough).

TL;DR: How can I tell the compressed size of a 64KB block of uncompressed data in a Zip file?

George Marques George Marques · Accepted Answer · 2016-08-09T23:12:14

I ended up making the Zip file "by hand" and compressing the blocks individually to calculate their hash.

#include <zlib.h>
#define BLOCK_SIZE 65536

z_stream strm;
// Using init2 to avold the zlib header
deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, -15, 8, Z_DEFAULT_STRATEGY);

int step = 0.

// len is the length of the whole input file
while (len - step > 0) {
    // Get the size of this block (either 64k or the remaining of the file if less)
    long block_size = (len - step) > BLOCK_SIZE ? BLOCK_SIZE : (len - step);

    // strm_in is the input buffer for the compression
    // skipped the buffer allocation for simplicity
    for (int i = 0; i < block_size; i++) {
        strm_in[i] = input_file_buffer[step + i];
    }

    // Calculate the hash (out of the scope for this question)
    // Store in a particular structure for reference later
    BlockData bd;
    bd.hash = make_block_hash(strm_in, block_size);

    // Update the zlib stream info
    // also skipped the out buffer allocation in this sample
    strm.avail_in = block_size;
    strm.avail_out = out_buffer_size;
    strm.next_in = strm_in;
    strm.next_out = strm_out;

    // Save the total bytes for comparison later
    int total_out_before = strm.total_out; 

    // Compress (assume the out buffer size will be always enough)
    deflate(&strm, Z_FULL_FLUSH); // A Full flush here is needed for the APPX format

    // Save the compressed block in the size
    compressed_file->save_buffer(strm_out, strm.total_out - total_out_before);

    // Save the size of the compressed block
    bd.compressed_size = strm.total_out - total_out_before;

    // Store the block data in a list
    blocks_info->push(bd);

    // Move to the next block
    step += block_size;
}

// End the compressed stream
strm.avail_in = 0;
strm.avail_out = out_buffer_size;
strm.next_in = strm_in;
strm.next_out = strm_out;

int total_out_before = strm.total_out;

// Finish the stream
deflate(&strm, Z_FINISH);

// Save the compression tail to the file
compressed_file->save_buffer(strm_out, strm.total_out - total_out_before);

// Clean up
deflateEnd(&strm);

After going all of this trouble I found a cross-platform open source tool that package and sign Windows store apps. Might be helpful for other people and also for me who thought signing outside of Windows would be impossible.

Determine the compressed block size with zlib/minizip

1 Answers