Methods to decrease Memory consumption in ArangoDB

3
votes

We currently have a ArangoDB cluster running on version 3.0.10 for a POC with about 81 GB stored on disk and Main memory consumption of about 98 GB distributed across 5 Primary DB servers. There are about 200 million vertices and 350 million Edges. There are 3 Edge collections and 3 document collections, most of the memory(80%) is consumed due to the presence of the edges

I'm exploring methods to decrease the main memory consumption. I'm wondering if there are any methods to compress/serialize the data so that less amount of main memory is utilized.

The reason for decreasing memory is to reduce infrastructure costs, I'm willing to trade-off on speed for my use case.

Please can you let me know, if there any Methods to reduce main memory consumption for ArangoDB

1
performancememory-managementarangodb
Why are you trying to reduce memory usage? Is it for speed? Resource availability? What kind of data do you have? Do you need the ability to query the largest portions of your data? What sort of data structures are you using? – Nate Gardner
@Nate : Thanks for the comments, i've updated the information on the question and made it more detailed. – pjesudhas
How much data are you storing on the edges? Is any of that data duplicated? – Nate Gardner

1 Answers

1
votes

It took us a while to find out that our original recommendation to set vm.overcommit_memory to 2 this is not good in all situations.

It seems that there is an issue with the bundled jemalloc memory allocator in ArangoDB with some environments.

With an vm.overcommit_memory kernel settings value of 2, the allocator had a problem of splitting existing memory mappings, which made the number of memory mappings of an arangod process grow over time. This could have led to the kernel refusing to hand out more memory to the arangod process, even if physical memory was still available. The kernel will only grant up to vm.max_map_count memory mappings to each process, which defaults to 65530 on many Linux environments.

Another issue when running jemalloc with vm.overcommit_memory set to 2 is that for some workloads the amount of memory that the Linux kernel tracks as "committed memory" also grows over time and does not decrease. So eventually the ArangoDB daemon process (arangod) may not get any more memory simply because it reaches the configured overcommit limit (physical RAM * overcommit_ratio + swap space).

So the solution here is to modify the value of vm.overcommit_memory from 2 to either 1 or 0. This will fix both of these problems. We are still observing ever-increasing virtual memory consumption when using jemalloc with any overcommit setting, but in practice this should not cause problems. So when adjusting the value of vm.overcommit_memory from 2 to either 0 or 1 (0 is the Linux kernel default btw.) this should improve the situation.

Another way to address the problem, which however requires compilation of ArangoDB from source, is to compile a build without jemalloc (-DUSE_JEMALLOC=Off when cmaking). I am just listing this as an alternative here for completeness. With the system's libc allocator you should see quite stable memory usage. We also tried another allocator, precisely the one from libmusl, and this also shows quite stable memory usage over time. The main problem here which makes exchanging the allocator a non-trivial issue is that jemalloc has very nice performance characteristics otherwise.

(quoting Jan Steemann as can be found on github)

Several new additions to the rocksdb storage engine were made meanwhile. We demonstrate how memory management works in rocksdb.

Many Options of the rocksdb storage engine are exposed to the outside via options.

Discussions and a research of a user led to two more options to be exposed for configuration with ArangoDB 3.7:

  • --rocksdb.cache-index-and-filter-blocks-with-high-priority
  • --rocksdb.pin-l0-filter-and-index-blocks-in-cache
  • --rocksdb.pin-top-level-index-and-filter