Decompress LZ4 stream of Unknown size

Question

I want to decompress a LZ4 raw stream inside binary data. I read the documentation and came up with the following function. According to the code, it should keep decompressing until it runs out of compressed data, but it doesn't. What am I doing wrong? Also, I want to add a writeln for the compressed size.

function Try(inStream: TMemorySTream): Boolean;
var
  i, j, k: int64;
  orig_buff: Pansichar;
  compressed_buff: Pansichar;
  lz4StreamDecode: PLZ4_streamDecode_t;
begin
  result := false;
  lz4StreamDecode := LZ4_createStreamDecode();
  k := 0;   
  while True do
  begin
    orig_buff := allocmem(100);
    compressed_buff := allocmem(4064);
    j := inStream.Read(orig_buff^, 100);
    i := lz4.LZ4_decompress_safe_continue(lz4StreamDecode, orig_buff,
      compressed_buff, j, 4064);
    Freemem(orig_buff);
    Freemem(compressed_buff);
    if i <= 0 then
      break;
    inc(k);
  end;
  if k <> 0 then
    result := True;
  LZ4_freeStreamDecode(lz4StreamDecode);    
end;

EDIT 1: Removed the part where it freed the output buffer (temp buffer), still the problem remains, instead of decompressing entire stream and then quitting, its just exiting the loop beforehand.

function Try(inStream: TMemorySTream): Boolean;
var
  i, j, k: int64;
  orig_buff: Pansichar;
  compressed_buff: Pansichar;
  lz4StreamDecode: PLZ4_streamDecode_t;
begin
  result := false;
  lz4StreamDecode := LZ4_createStreamDecode();
  compressed_buff := allocmem(64 * 1024 * 1024);
  k := 0;   
  while True do
  begin
    orig_buff := allocmem(100);
    j := inStream.Read(orig_buff^, 100);
    i := lz4.LZ4_decompress_safe_continue(lz4StreamDecode, orig_buff,
      compressed_buff, j, (64 * 1024 * 1024));
    Freemem(orig_buff);
    if i <= 0 then
      break;
    inc(k);
  end;
  if k <> 0 then
    result := True;
  Freemem(compressed_buff);
  LZ4_freeStreamDecode(lz4StreamDecode);    
end;

Wouldn't it be better to know the length? Isn't that the intention. — David Heffernan
You are going about it the wrong way. Compress the data and store the compressed size and the data. — David Heffernan
That would be the case if I was compressing the data, this is to decompress any LZ4 stream embedded in a dump, so am practically running this on every offset. — Dremora

Dsm Dsm · Accepted Answer · 2017-03-08T13:45:20

LZ4_Decompress_safe_continue uses the 'previous memory block' because the compression also relied on that. You are destroying that memory block. You must not do that. Instead you should use two buffers and switch between them. You should create the buffers before you start and only destroy them when you are finished at the end of the decompression. You say you have read the manual. I don't think you have. It gives an explicit example for doing this and also explains how it works.

Edit

This is how I would modify your function. Sorry - unable to test

Note I am using 2 buffers and toggling between them. This is one of the things that you are missing

Also I wouldn't use Try as a function name, it is a reserved word!

function XTry(inStream: TMemorySTream): Boolean;
var
  i, j, k: int64;
  orig_buff: Pansichar;
  compressed_buff1, compressed_buff2: Pansichar;
  lz4StreamDecode: PLZ4_streamDecode_t;
begin
  result := false;
  lz4StreamDecode := LZ4_createStreamDecode();
  orig_buff := allocMem( 128 *1024 );
  compressed_buff1 := allocmem(1024 * 1024);
  compressed_buff2 := allocmem(1024 * 1024);
  k := 0;   
  while True do
  begin
    j := inStream.Read(orig_buff^, 128*1024);
    i := lz4.LZ4_decompress_safe_continue(lz4StreamDecode, orig_buff,
      compressed_buff1, j, 4096);  // <<<< Note 1st buffer
    if i <= 0 then
      break;
    inc(k);
    j := inStream.Read(orig_buff^, 128*1024);
    i := lz4.LZ4_decompress_safe_continue(lz4StreamDecode, orig_buff,
      compressed_buff2, j, 4096);   // Note 2nd buffer
    if i <= 0 then
      break;
    inc(k);
  end;
  if k <> 0 then
    result := True;
  LZ4_freeStreamDecode(lz4StreamDecode);    
  Freemem(compressed_buff1);
  Freemem(compressed_buff2);
  Freemem(orig_buff);

end;

Obviously this can be cleaned up a lot - I just wanted to keep as close to your original code as possible

Decompress LZ4 stream of Unknown size

1 Answers

Edit