Zlib Compression differences between zlib.c Adler implemention and Deflate, Ionic Deflate

Question

I have been experiencing some issues with imcompatiabilities between the difference implementations of zlib compression.

As a test case I thought to create test data with 10000 doubles ranging from 0 - 10000.

I created some test code to compress and decompress this data that uses the compress and uncompress in zlib.c

unsigned int Test(char* comparisonFile)
{
    unsigned long partsSize = 0x80000;
    const int arraySize = 10000;
    Bytef doubleArray[sizeof(double) * arraySize];
    Bytef outBuffer[sizeof(double) * arraySize];

    for (int i = 0; i < arraySize; i++)
    {
        Bytef doubleBytes[sizeof(double)];
        *(double*)(doubleBytes) = (double)i;
        for (int x = 0; x < 8; x++)
            doubleArray[(8 * i) + x] = doubleBytes[x];
    }
    compress(outBuffer, &partsSize, doubleArray, sizeof(double) * arraySize);

    //create file of compressed data
    char * filename = "zlibCompressed";
    FILE * file = fopen(filename, "w+b");
    int compressResult = et_int64(fwrite((char *)outBuffer, 1, size_t(partsSize), file));
    fclose(file);

    //load  file of compressed data either from zlib or other
    if (comparisonFile != NULL)
        filename = comparisonFile;
    FILE * compressedFile = fopen(filename, "r+b");
    if (compressedFile == NULL)
        return -1;

    unsigned long outBufferSize = sizeof(double) * arraySize;
    fseek(compressedFile, 0, SEEK_END);
    long fsize = ftell(compressedFile);
    fseek(compressedFile, 0, SEEK_SET);  /* same as rewind(f); */
    partsSize = int(fsize);

    double * doubleResult = new double [arraySize];
    Bytef* inBuffer = (Bytef*)malloc(sizeof(Bytef)*partsSize);
    int readresult = et_int64(fread((char *)inBuffer, 1, partsSize , compressedFile));

    if (readresult != partsSize)
        return -1;

    Bytef *  uncompressedOutBuffer = static_cast<Bytef*>((void *)doubleResult);
    int result = uncompress(uncompressedOutBuffer, &outBufferSize, inBuffer, partsSize);

    for (int i = 0; i < arraySize; i++)
    {
        // uncompressed data does not match expectation
        if ((int)doubleResult[i] != i)
            return -2;
    }
    fclose(compressedFile);
    return 0;

}

This allows me to test the internal compression and also substitute compression results from C#.

However, when I use ionic or standard deflate in the following manner I can only recover about 8150 of the expected 10000.

The uncompression returns a:

Z_DATA_ERROR

Given that these seem in theory interoperable, i'm not sure why C# compression results can only be partially unpacked with Adler's zlib? Any help out there?

public void ZlibTest()
        {
            byte[] buffer;
            using (var ms = new MemoryStream())
            {
                for (int i = 0; i < 10000; i++)
                    ms.Write(BitConverter.GetBytes((double) i), 0, sizeof(double));
                buffer = ms.ToArray();
            }

            var file = "dummy1";

            if (File.Exists(file))
                File.Delete(file);


            using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
            {
                using (var resultStream = new MemoryStream())
                {
                    using (var compressionStream2 = new Ionic.Zlib.ZlibStream(resultStream, Ionic.Zlib.CompressionMode.Compress, CompressionLevel.Default))
                    {
                        compressionStream2.Write(buffer, 0, buffer.Length);
                        var packetLength = (int)resultStream.Length;
                        fs.Write(resultStream.ToArray(), 0, packetLength);
                    }
                }
            }

            file = "dummy2";
            using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
            {
                using (var resultStream = new MemoryStream())
                {
                    using (var compressionStream2 = new System.IO.Compression.DeflateStream(resultStream, System.IO.Compression.CompressionMode.Compress))
                    {
                        compressionStream2.Write(buffer, 0, buffer.Length);
                        var packetLength = (int)resultStream.Length;
                        fs.Write(BitConverter.GetBytes((ushort)40056), 0, sizeof(ushort));
                        fs.Write(resultStream.ToArray(), 0, packetLength);
                    }
                }
            }
        }

Mark Adler Mark Adler · Accepted Answer · 2019-03-08T23:27:14

They're both using the same zlib. There must be an error in your C# code. It is possible that you are not reading the file in binary mode in C#, which would cause occasional corruption.

Zlib Compression differences between zlib.c Adler implemention and Deflate, Ionic Deflate

1 Answers