Stream adapter to provide write-to interface to write-out utility, but also supply read-out interface

Question

I don't like using MemoryStream objects inbetween stream interfaces. They are awkward, requiring you to re-seek to the start, and will also peak memory usage in demanding situations.

Sometimes a utility will only work a certain way. Perhaps it will output byte[]s, or write to a stream, or is a stream in a pipeline that you read from, pulling the data through.

This Newtonsoft JSON serializer is a utility which can only write to a stream.

var js = new Newtonsoft.Json.JsonSerializer();
var sw = new StreamWriter(ps);
js.Serialize(sw, o);

This is a problem for me, because I want to chain:

• IEnumerable
• JSON serialization
• GZIP compression
• HTTP to client
• (Network)
• HTTP from Server
• GZIP decompression
• JSON deserialization
• IEnumerable

Apart from the difficulties getting the JSON deserializer to present a nice IEnumerable interface, the rest of the parts don't provide an interface suitable for pipelining. Even the GZIP compression side is the wrong way around.

Ideally, on the server-side I would be able to do:

IEnumerable<object> o = GetData(); 
var js = new Newtonsoft.Json.JsonSerialization(o);
var gz = new System.IO.Compression.GZipStream(js, System.IO.Compression.CompressionMode.Compress, true);
return new FileStreamResult(gz, "application/x-gzip");

I could possibly extend the Newtonsoft project to provide a pipeline implementation, and I may do so. But until then I need a solution, and I believe one is required for other utilities (including the BCL GZipStream).

1. Are there any solutions which allow one to join such utilities more efficiently?
2. Is there a library which contains an adapter for such situations?

I am working on such a library, not expecting there to be such a library.

Answer 1

The answer is the brand new StreamAdaptor project: https://bitbucket.org/merarischroeder/alivate-stream-adaptor. It still needs a bit of work - would be nice to package it as a NuGet package, but it's all there and tested.

So the interface will look a bit like this:

var data = GetData(); //Get the source data
var sa = new StreamAdaptor(); //This is what wraps the write-only utility source
sa.UpstreamSource((ps) => //ps is the dummy stream which does most of the magic
{
    //This anon. function is run on a separate thread and can therefore be blocked
    var sw = new StreamWriter(ps);
    sw.AutoFlush = true;
    var js = new Newtonsoft.Json.JsonSerializer();
    js.Serialize(sw, data); //This is the main component of the implementation
    sw.Flush();
});

var sa2 = new StreamAdaptor();
sa2.UpstreamSource((ps) =>
{

    using (var gz = new System.IO.Compression.GZipStream(ps, System.IO.Compression.CompressionMode.Compress, true))
        sa.CopyTo(gz);
});

The reverse process is easier with natural support for a read-through pipeline

System.IO.Compression.GZipStream sw = new System.IO.Compression.GZipStream(sa2, System.IO.Compression.CompressionMode.Decompress);
var jsonTextReader = new JsonTextReader(new StreamReader(sw));
return TestA.Deserialize(jsonTextReader);

I also demonstrate there a workaround to the IEnumerable<> deserializing issue. It requires you to create your own deserializer leveraging JsonTextReader, but it works well.

The serializer supports IEnumerable natively. The GetData function above, sets up the data source for the serializer using IEnumerable functions (among other things):

public static IEnumerable<TestB> GetTestBs()
{
    for (int i = 0; i < 2; i++)
    {
        var b = new TestB();
        b.A = "A";
        b.B = "B";
        b.C = TestB.GetCs();

        yield return b;
    }
}

It's Deserialisation which requires a workaround. Keep in mind that IEnumerable<> properties need to be listed all at the end of the JSON stream/objects, because enumeration is deferred, yet JSON deserialization is linear.

The Deserialization entry point:

public static TestA Deserialize(JsonTextReader reader)
{
    TestA a = new TestA();

    reader.Read();
    reader.Read();
    if (!reader.Value.Equals("TestBs"))
        throw new Exception("Expected property 'TestBs' first");
    reader.Read(); //Start array
    a.TestBs = DeserializeTestBs(reader); //IEnumerable property last
    return a;
}

One of the IEnumerable deserializer functions:

static IEnumerable<TestB> DeserializeTestBs(JsonTextReader reader)
{
    while (reader.Read())
    {
        if (reader.TokenType == JsonToken.EndArray)
            break;
        yield return TestB.Deserialize(reader);
    }
    reader.Read(); //End of object
}

This can of course be achieved with trial and error, although built-in support in JSON.NET is desirable.