Memory Limitations (MarkLogic)

2
votes

I am currently trying to see if there are options to increase the overall memory limit of MarkLogic. I have attempted to increase the memory options in admin console but there seems to be no increase in it's limit. I looked through various forums and came across this thread. Stating that

"When the total hash join table size of all running SPARQL queries exceeds 50% of the host memory ... canceled with the "XDMP-MEMCANCELED" error"

So the question is can I increase the host memory % limit to say 90% for instance?

At the same time it also states that I can increase my memory to host to improve this. Do host memory refer to virtual memory? I tried increasing it but the memory limit when the error message is generated remains same.

System is currently running Windows.

===Update=== SPARQL Query Used

SELECT *
WHERE{
?people  </posted>  ?question .
?question </has_comment> ?comment .
?question </date_posted> ?question_date_posted .
?question </has_tags> ?tags .
?commented_by_person </commented_by> ?comment .
?comment </date_commented> ?comment_date_commented .
?comment </text> ?comment_text .
?people </from> ?school .
?people </has_age> ?age
}

Error Generated by Console

XDMP-MEMCANCELED: { SELECT people ... ?comment . } } -- Canceled because of memory usage on host desktop-60k8s55, requestMemory=4359540880, totalMemory=4359540880, memoryLimit=4294967296

1
marklogicmarklogic-9
It's helpful to add additional context to your question, because it sounds like you have encountered an issue that you believe is memory related. That may well be true, there are also coding issues that will overtax memory. Are you seeing MEMCANCELED or other issues being logged by the server. – Michael Gardner
I have added the error message I hope it helps. Do let me know if it is not the correct message you are looking for. Right now I am currently trying to do lots of joins on purpose to see how much MarkLogic can handle and the time taken to execute these. Therefore, I am currently trying to search for ways to increase memory. @Michael Gardner – WhiteSolstice
I am currently running it on a local system @Michael Gardner – WhiteSolstice
Sorry but I am currently running it on a windows OS. I will definitely have a look at that if I ever touch a linux system. @Michael Gardner – WhiteSolstice

1 Answers

3
votes

MarkLogic memory consumption is a function of the various group and database cache settings, as well as the number of active forests, among other things.

I suggest checking out the Fundamentals of Resource Consumption paper, which has a section Memory Utilization in MarkLogic Server. It's also worthwhile to check out the Query and Tuning Guide

Host memory refers to the amount of RAM on the configured system or instance.

You can increase the amount of memory that MarkLogic will potentially consume by making changes to the group level settings.

In the Admin UI, go to the Configure -> Groups -> [Default] Configure tab, and ensure the 'cache sizing' is set to Manual and then you can adjust the various cache sizes as wanted.

You will also want to make sure that you have already done the appropriate OS level tuning for swap sizes and other best practices for the OS you are on.

If you later add memory to the system, then you will also need to make sure that you go back to adjust the Group caches and the OS swap.