ZLIB compression

0
votes

I'm working on a project wherein if something goes wrong on my disk I'll send chunks of memory to another disk after compressing it using ZLIB. This dump then I plan to download and use for further debugging. This compression and uploading is to be done one chunk at a time - say 1024 K.

The catch is I need to have output chunks of exactly 1024K before I upload it to the other disk (besides this there will be headers and trailers as well so this chunk size is the size of the compressed data) as then I can easily decompress them and use it. I tried changing the put_byte function such that once the output buffer is 1024K I can send it over to the other disk but this does not seem to be working.

I thought of checking for Z_BUF_ERROR and then sending it over and restarting the entire compression procedure but then I would not be sure as to whether all the pending input / output have been flushed.

Is there some other way I can go about doing this - i.e. ensuring that the packets are sent when the output buffer (compressed data size) has exactly 1024K bytes.

1
ccompressionzlib
O/P or OP => output buffer storing the compressed data IP or I/P => input buffer storing the raw data. – Zshn
If you simply want to break down the output of deflate() into 1K chunks, then just provide deflate() with 1K of output, i.e. strm.avail.out = 1024;. Provide input to deflate() until strm.avail.out == 0 or you run out of input. Then write out that chunk. – Mark Adler
@MarkAdler : Hi Mark, I made the following changes to put_byte in the ZLIB library :- Whenever a character is copied into the pending buffer I copy the same into my output buffer of size 63K (this is different from the buffer pointed to by next_out). I made the necessary changes in flush_pending() too. The problem is when I run inflate on the compressed data I get the following error :- "invalid or incomplete deflate data". I have made no changes to the window sizes. Still why am I ending up with this error ?? – Zshn
With no code I have no idea what you're doing. – Mark Adler

1 Answers

0
votes

Maybe you would have better luck using a higher-level facade library like the C++ http://github.com/rudi-cilibrasi/zlibcomplete ZLibComplete library. It uses std::string and requires no dynamic allocation so you don't need to manage any buffers yourself simply put in std::string and write out the returned std::string You also don't need to check any return values nor do you need to loop conditionally. It simply works in terms of strings: put a string in, get a string out as long as you like for compression or decompression. There are short working example programs that you can start with for both common compression formats, GZip and ZLib.