zlib Z_BUF_ERROR with specific file and specific buffer sizes

Question

I'm developing some code that needs to be capable of unzipping large gzip'd files (up to 5GB uncompressed) and reading them into memory. I would prefer to be clean about this and not simply unzip them to disk temporarily so I've been working with zlib to try to accomplish this. I've got it running, most of the way. Meaning it runs for 4 of the 5 files I've used as input. The other file gives a Z_BUF_ERROR right in the middle of processing and I'd prefer not to ignore it.

This initially happened in different code but eventually I brought it all the way back to the example code that I got from zpipe.c on the zlib web page, and no matter what code I used, it resulted in the same Z_BUF_ERROR and only with this file. I played with the code for quite a while after reading several posts about Z_BUF_ERROR and after reading the manual on this as well. Eventually I was able to find a way to make it work by changing the size of the buffer used to hold the inflated output. Normally at this point I'd call it a day until it reported an error with another file, but ideally this will be production level code at some point and I'd like to understand what the error is so I can prevent it rather than just fix it for now. Especially since gzip is able to compress and decompress the file just fine.

I've tried this with the following variations:

different platforms: CentOS, OSX
different versions of zlib: 1.2.3, 1.2.8 (same results)
values of CHUNK and the number of bytes output (complete is 783049330):
- 2000000: 783049330
- 1048576: 783049330
- 1000000: 783049330
- 100000: 783049330
- 30000: 248421347
- 25000: 31095404
- 20000: 783049330
- 19000: 155821787
- 18000: 412613687
- 17000: 55799133
- 16384: 37541674
- 16000: 783049330
any CHUNK size greater than 4100000 gives an error
tried declaring out with a value greater than CHUNK (same results)
tried using malloc to declare out (same results)
tried using gzip to uncompress and then compress the file again thinking something may have been off in the gzip metadata (same results)
tried compressing a separate uncompressed version of the file using gzip for the same purpose, but I believe the original .gz file was created from this one (same results)

I may have tried a few things outside of this list as I've been trying to get to the bottom of it for a while, but only changing the CHUNK size will make this work. My only concern is that I don't know why a different size will work and I'm worried that another CHUNK size will put other files at risk for this issue, because again, this is only an issue for one file.

` CODE:

FILE* fp = fopen( argv[1], "rb" );
int ret = inf( fp, stdout );
fclose( fp );

int inf(FILE *source, FILE *dest)
{
  size_t CHUNK = 100000;
  int count = 0;
  int ret;
  unsigned have;
  z_stream strm;
  unsigned char in[CHUNK];
  unsigned char out[CHUNK];
  char out_str[CHUNK];

  /* allocate inflate state */
  strm.zalloc = Z_NULL;
  strm.zfree = Z_NULL;
  strm.opaque = Z_NULL;
  strm.avail_in = 0;
  strm.next_in = Z_NULL;
  ret = inflateInit2(&strm, 16+MAX_WBITS);
  if (ret != Z_OK)
    return ret;
  /* decompress until deflate stream ends or end of file */
  do {
    strm.avail_in = fread(in, 1, CHUNK, source);
    if (ferror(source)) {
      (void)inflateEnd(&strm);
      return Z_ERRNO;
    }
    if (strm.avail_in == 0)
      break;
    strm.next_in = in;

    /* run inflate() on input until output buffer not full */
    do {
      strm.avail_out = CHUNK;
      strm.next_out = out;
      ret = inflate(&strm, Z_NO_FLUSH);
      switch (ret) {
        case Z_NEED_DICT:
          ret = Z_DATA_ERROR;     /* and fall through */
        case Z_DATA_ERROR:
        case Z_MEM_ERROR:
          (void)inflateEnd(&strm);
          return ret;
      }
      have = CHUNK - strm.avail_out;
      char out_str[have+1];
      strncpy( out_str, (char*)out, have );
      out_str[have] = '\0';

      // testing the ability to store the result in a string object and viewing the output
      std::cout << "out_str: " << std::string(out_str) << " ::" << std::endl;

      if( ret == Z_BUF_ERROR ){
        std::cout << "Z_BUF_ERROR!" << std::endl;
        exit(1);
      }
    } while (strm.avail_out == 0);

    /* done when inflate() says it's done */
  } while (ret != Z_STREAM_END);

  /* clean up and return */
  (void)inflateEnd(&strm);
  return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
}

`

Mark Adler Mark Adler · Accepted Answer · 2014-10-22T02:51:01

You should read the commentary where you got that code from. Z_BUF_ERROR is just an indication that there was nothing for inflate() to do on that call. Simply continue and provide more input data and more output space for the next inflate() call.

zlib Z_BUF_ERROR with specific file and specific buffer sizes

1 Answers