DeflateStream: compress files while reading

Question

I'm trying to send a big chunk of data through WCF (some GBs). I would like to compress the file while reading it using Streams, but looks like DeflateStream has two modes:

• Compress (writes on the stream)
• Decompress (reads the stream)

None of these modes works in my case. I would like to read an un-compressed file from disk and return a compressed stream through WCF.

Is there a way to do so, or I have to use a temporal file (or MemoryStream)?

Is a missing feature or is just not possible for some reason?

Answer 1

It looks like you are trying to compress while reading the file. The way deflatestream is written, compression has to happen as part of a write. Try wrapping the stream that you are sending over the wire, not the stream that you are reading off disk. If they are the same, you need an intermediate stream.

Answer 2

Try using these methods for compressing and decompressing a byte array.

    private static byte[] Compress(byte[] data)
    {
        byte[] retVal;
        using (MemoryStream compressedMemoryStream = new MemoryStream())
        {
            DeflateStream compressStream = new DeflateStream(compressedMemoryStream, CompressionMode.Compress, true);
            compressStream.Write(data, 0, data.Length);
            compressStream.Close();
            retVal = new byte[compressedMemoryStream.Length];
            compressedMemoryStream.Position = 0L;
            compressedMemoryStream.Read(retVal, 0, retVal.Length);
            compressedMemoryStream.Close();
            compressStream.Close();
        }
        return retVal;
    }



    private static byte[] Decompress(byte[] data)
    {
        byte[] retVal;
        using (MemoryStream compressedMemoryStream = new MemoryStream())
        {
            compressedMemoryStream.Write(data, 0, data.Length);
            compressedMemoryStream.Position = 0L;
            MemoryStream decompressedMemoryStream = new MemoryStream();
            DeflateStream decompressStream = new DeflateStream(compressedMemoryStream, CompressionMode.Decompress);
            decompressStream.CopyTo(decompressedMemoryStream);
            retVal = new byte[decompressedMemoryStream.Length];
            decompressedMemoryStream.Position = 0L;
            decompressedMemoryStream.Read(retVal, 0, retVal.Length);
            compressedMemoryStream.Close();
            decompressedMemoryStream.Close();
            decompressStream.Close();
        }
        return retVal;
}

Answer 3

You should have something like:

public void CompressData(Stream uncompressedSourceStream, Stream compressedDestinationStream)
{
    using (DeflateStream compressionStream = new DeflateStream(compressedDestinationStream, CompressionMode.Compress))
    {
        uncompressedSourceStream.CopyTo(compressionStream);
    }
}

public void DecompressData(Stream compressedSourceStream, Stream uncompressedDestinationStream)
{
    using (DeflateStream decompressionStream = new DeflateStream(uncompressedDestinationStream, CompressionMode.Decompress))
    {
        compressedSourceStream.CopyTo(decompressionStream);
    }
}

using (FileStream sourceStream = File.OpenRead(@"C:\MyDir\MyFile.txt))
using (FileStream destinationStream = File.OpenWrite(@"C:\MyDir\MyCompressedFile.txt.cp"))
{
    CompressData(sourceStream, destinationStream)
}

Also, be aware that you may have to change the WCF settings in your application's .config file to allow really large things to transfer.

Answer 4

You can wrap the DeflateStream in a stream of your own. Everytime you want to read from the compressing stream, you have to feed bytes into the deflatestream, until it writes to a buffer. You can then return bytes from that buffer.

public class CompressingStream : Stream
{
    private readonly DeflateStream _deflateStream;
    private readonly MemoryStream _buffer;
    private Stream _inputStream;
    private readonly byte[] _fileBuffer = new byte[64 * 1024];

    public CompressingStream(Stream inputStream)
    {
        _inputStream = inputStream;
        _buffer = new MemoryStream();
        _deflateStream = new DeflateStream(_buffer, CompressionMode.Compress, true);
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        while (true)
        {
            var read = _buffer.Read(buffer, offset, count);

            if (read > 0) return read;

            if (_inputStream == null) return 0;

            _buffer.Position = 0;
            read = _inputStream.Read(_fileBuffer, 0, _fileBuffer.Length);
            if (read == 0)
            {
                _inputStream.Close();
                _inputStream = null;
                _deflateStream.Close();
            }
            else
            {
                _deflateStream.Write(_fileBuffer, 0, read);
            }
            _buffer.SetLength(_buffer.Position);
            _buffer.Position = 0;
        }
    }

    public override bool CanRead
    {
        get { return true; }
    }
#region Remaining overrides...
}

Whenever wcf reads from the stream, the compressing stream will write to compressing DeflateStream, until it kan read from the output buffer (_buffer). It's ugly, but it works.

Answer 5

I was trying to create a Stream that, whenever Read is invoked:

• Reads a chunk of data from the source file
• Writes the chunk on a DeflateStream connected to a MemoryStream
• Copies the content of the MemoryStream to the Read buffer parameter.

Of course, it will be more difficult since the size of both streams are not similar.

At the end I've dismissed this option since I find no way to predict the size of the resulting compressed file without fully compressing it at the beginning.

Reading the file, however, is able to predict the file size so maybe with another implementation of DeflateStream could be possible.

Hope it helps other lost souls out there...

Answer 6

Azure API for blobs has an alternative for UploadStream(stream). You can get the stream with OpenWrite(). So now you are in control of pushing bytes and therefor can compress while streaming content to the service

using (var uploadStream = blob.OpenWrite())
using (var deflateStream = new DeflateStream(uploadStream, CompressionMode.Compress))
{
    stream.CopyTo(deflateStream);
}

I haven't check the WCF API but I would be surprised if you cannot do the same.