Large byte array in Windows Store App

Question

I am now writing a Windows Store App Reader that is used for viewing PDF files using C#.

For now, I am trying to solve a problem that might be causing memory leak. In app, there is a method for encrypting and decrypting the PDF files, encryption/decryption method is custom method and the PDF files' size are between 1MB and 120MB.

During encrypt and decrypt process, the file must be read into memory as byte array, and processes the bytes for encryption and decryption. After returning from the method, the byte array is out of scope. But it seems that because it is a large byte array(>85k), I thought it is declared in the large object heap (LOH) and after getting garbage collected, it looks like LOH becomes fragmented.

And after the app opens, views and switching more PDF files, the memory usage is getting higher and at some points out of memory exception occurs.

Is there a way to solve large byte array problem? Or is anyone encounter such a problem?

I have found Memory Mapped Files class but it cannot be used in Windows Store App.

I have also tried with chunk of byte array(<85k) and process the file part by part, but it takes too long for encryption and decryption before viewing the PDF.

Below is the code snippet:
public static async Task CryptoFile(StorageFolder Folder, string FileName, bool IsReaderFile)
{
...
...

byte[] fileContent = null;
buffer = await FileIO.ReadBufferAsync(file);
using (DataReader dataReader = DataReader.FromBuffer(buffer))
{
        fileContent = new byte[dataReader.UnconsumedBufferLength];
    dataReader.ReadBytes(fileContent);
}

if (fileContent == null)
        CryptResult = false;
else if (fileContent.Length == 0)
        CryptResult = false;

if (CryptResult)
{
        //Encrypt/decrypt file
    fileContent = await RecodeFile(fileContent, CryptKey, IsReaderFile);

    //Delete the original file
    file = await Folder.GetFileAsync(FileName);
    await file.DeleteAsync(StorageDeleteOption.PermanentDelete);

    //Recreate file
    file = await Folder.CreateFileAsync(FileName);
    using (IRandomAccessStream fs = await file.OpenAsync(FileAccessMode.ReadWrite))
    {
            using (IOutputStream outStream = fs.GetOutputStreamAt(0))
        {
                using (DataWriter dataWriter = new DataWriter(outStream))
            {
                    dataWriter.WriteBytes(fileContent);
                await dataWriter.StoreAsync();
                dataWriter.DetachStream();
            }

            await outStream.FlushAsync();
        }
    }

    ....
    }

}

fileContent is the byte array for reading the whole file content and it will be out of scope when return from the method, and I think fileContent is causing the memory fragmentation because of its size it might be declared in LOH.

Are you positive there aren't any roots that are keeping your memory high? What are you doing with the byte array once it's finished decrypting? Can you give us some code or at least a process flow on how it all happens? — Simon Whitehead

Richard Smith Richard Smith · Accepted Answer · 2013-06-27T07:12:42

Assuming your issue is related to the LOH have you considered wrapping up the smaller arrays into a LOH friendly array substitute. You can optimise the example further if you are willing to constrain the block sizes to powers of two. The div mod then becomes a shift and a bitmask. Obviously you can implement all the ICollection stuff if needed.

I've used this approach before and the performance is reasonable.

public sealed class ArrayRope<T>
{
    private readonly T[][] map;

    private readonly int blockSize;
    private readonly int noOfBlocks;

    public ArrayRope(int blockSize, int noOfBlocks)
    {
        ...
    }

    public T this[int index]
    {
        get
        {
            int ropeIndex = index / blockSize;
            int offset = index % blockSize;

            return map[ropeIndex][offset];
        }
        set
        {
            int ropeIndex = index / blockSize;
            int offset = index % blockSize;

            map[ropeIndex][offset] = value;
        }
    }

    ...
}

Large byte array in Windows Store App

1 Answers