"Streaming" compression/decompression by chunk

1
votes

I'm working on a JavaScript compression library. I've already created a relatively fast DEFLATE compressor and decompressor, but they require the data to be fully loaded in memory prior to use. I don't think adding streaming support for the compression should be too difficult; I can just compress whatever data is available into a set of full blocks, append the result to the output stream, and avoid setting the BFINAL marker until the final chunk is passed. However, an issue arises with decompression streams.

Since my code currently needs to read a full chunk to generate an output and does not preserve state, the best I can do is guess how long a chunk is and hope that I reach the end, because if I don't, I've wasted multiple CPU cycles reading the headers, Huffman codes, and length/literal and distance codes of the unfinished block, and I'll have to do that all over again. This seems to be an awful way of solving the problem, and I'm wondering if there's any other way to do this that doesn't involve rewriting my code to preserve state.

Current decompression code

1
javascriptstreamcompressiondeflatechunking
Are you doing this in-browser? If so, why not let the browser handle this for you? – Brad
The browser doesn't actually have compression or decompression APIs for JS, despite supporting compressed Content-Encoding. The closest thing we have is Chrome-specific, in draft stage, and slower than my library for fully-loaded data: wicg.github.io/compression – 101arrowz

1 Answers

1
votes

Nope, you'll need to preserve state. Or be prepared to read the entire stream into memory. There is nothing that prevents a deflate stream from consisting of a single long deflate block.