zlib, how to decompress stream of compressed data chunk?

Question

I have a problem about transfer compressed data across network.

The data size is around a few hundred MBs. My plan is to divide the data into 1MB chunk and compress each chunk with zlib, then stream the compressed data over network. On the other end of the network, data will be decompressed by zlib.

My question is that since I stream the compressed data, there won't be information about where each compressed chunk starts and ends in the stream. I am not sure if zlib can decompress such compressed data stream.

If zlib can, please kindly let me know what flush mode I should use in deflate/inflate methods?

Thanks!

Answer 1

It is not clear why you are dividing the data into chunks, or why you would need to do any special flushing. If you just mean feeding the data to zlib in chunks, that is how zlib is normally used. zlib doesn't care how you feed it it the data -- big chunks, little chunks, one byte at a time, one giant chunk, etc., will not change the compressed result.

Flushing does change the compressed result, degrading it slightly or significantly depending on how often you flush and how you flush.

Flushing is used when you want to assure that some portion of the data is fully received at a known boundary in the compressed data, or if you want to be able to recover part of the data if not all of it is received.

Answer 2

If the strategy you used is a must, you could make a protocol between your host and remote, such as:

02 123456 the-compressed-data 654321 the-compressed-data

The 3 numbers are: 1. the number of chunks of data, here is 2 chunks 2. the bytes of first chunk 3. the bytes of second chunk respectively.