Saxon memory not being released

Question

I'm having issues releasing the memory Saxon is using for loading the file into memory. This is a reduced version of the code we have:

Task newTask = Task.Run(() =>
{
    Processor processor = new Processor();
    DocumentBuilder documentBuilder = processor.NewDocumentBuilder();
    documentBuilder.IsLineNumbering = true;
    documentBuilder.WhitespacePolicy = WhitespacePolicy.PreserveAll;
    XQueryCompiler compiler = processor.NewXQueryCompiler();
    string query = BuildXqueryString();

    if (!String.IsNullOrEmpty(query))
    {
        XQueryExecutable executable = compiler.Compile(query);
        XQueryEvaluator evaluator = executable.Load();
        evaluator.ContextItem = documentBuilder.Build(xmlNode);
        var evaluations = evaluator.Evaluate();
    }
}

We are running the code inside a Task because we are having lots of evaluations happening at the same time, many of them with XML files up to 2GB in size. For this reason we need the memory used by Saxon during the execution of the Build() method released as soon as possible, but what is happening is that the memory is not completely reclaimed by the garbage collector. After each evaluation the memory usage rises and rises, and is never completely freed, so with a couple of these big evaluations the memory usage rises to its max. I've tried calling the Dispose() method of Task, setting all the variables to null, but nothing seems to work. Any ideas what could be happening?

Edit:

Thank you all for your answers. I realized that the issue was actually not in the Saxon code, but instead with a XmlDocument that wasn't releasing the memory. Sorry for the confusion.

Do the XQueryXXX classes have a dispose? that would be the ones you must destroy to release the memory. Also, maybe a GC.Collect() can help with this. — Gusman
What version of Saxon are you using? I have created a bug issue for this on our site: saxonica.plan.io/issues/2746 Is it possible for you to send the full code so that we can reproduce this and investigate further. You can send the code privately or attach it to the bug issue. — ond1
Are you using Saxon-EE with bytecode generation? This can cause memory problems because every time you compile a query, new classes are generated, and .NET typically (to simplify a complex story) does not free the memory occupied by dynamically loaded classes. — Michael Kay

Siraj Mansour Siraj Mansour · Accepted Answer · 2016-05-13T01:07:23

Check if any of the classes of which instances are used inside the task are disposable, they could be holding resources until dispose is called explicitly (after being done with their job). also checkout using blocks.

Also noted from the official documentation:

The first thing the application needs to do is to create a Processor. The Processor holds configuration information for Saxon, and shared resources such as the name pool and schema pool. It is possible to run multiple processors concurrently if required, but it is usually more economical for all Saxon processes within a single application to use the same Processor.

so how about using the same processor instance for all your tasks ? it could help in case it uses cached resources.

Saxon memory not being released

1 Answers