Change the encoding to UTF-8 on a stream (MemoryMappedViewStream)

Question

I am using the code below to read a ~2.5Gb Xml file as fast as I can (thanks to MemoryMappedFile). However, I am getting the following exception: "'.', hexadecimal value 0x00, is an invalid character. Line 9778, position 73249406.". I beleive it is due to some encoding problem. How do I make sure that the MemoryMappedViewStream reads the file using UTF-8?

static void Main(string[] args)
{
    using (var file = MemoryMappedFile.CreateFromFile(@"d:\temp\temp.xml", FileMode.Open, "MyMemMapFile"))
    {
        using (MemoryMappedViewStream stream = file.CreateViewStream())
        {
            Read(stream);
        }
    }
}

static void Read(Stream stream)
{
    using (XmlReader reader = XmlReader.Create(stream))
    {
        reader.MoveToContent();

        while (reader.Read())
        {
        }
     }
 }

No, it is because you ran off the end of the mapping. You can only ever hope to map the full 2.5 gigabytes on a 64-bit operating system. This code doesn't accomplish anything, you are actually making it slower by copying the data twice. First to the file system cache, again to the view. Memory mapped files are only useful if you read the same data from them repeatedly. You don't. — Hans Passant

Hans Hans · Accepted Answer · 2011-08-19T17:40:43

You could use the StreamReader class to set the encoding:

static void Main(string[] args)
{
  using (var file = MemoryMappedFile.CreateFromFile(@"d:\temp\temp.xml", FileMode.Open,  "MyMemMapFile"))
  {
     using (MemoryMappedViewStream stream = file.CreateViewStream())
    {
        Read(stream);
    }
   }
}

static void Read(Stream stream)
{
  using (XmlReader reader = XmlReader.Create(new StreamReader(stream, Encoding.UTF8)))
  {
     reader.MoveToContent();

    while (reader.Read())
    {
    }
 }
}

Hope, this helps.

Change the encoding to UTF-8 on a stream (MemoryMappedViewStream)

2 Answers