ArangoDB: Is it bad if data or index does not fit in RAM anymore?

Question

Dear ArangoDB community,

I am wondering if it is bad when we use ArangoDB, and if unfortunately the data + index is grown too large and does not fit anymore in the RAM. What's happening then? Does it ruin the system performance horribly?

For TokuMX, which is a very fascinating Fork of MongoDB (TokuMX offers ACID Transactions which I need), they say thats NO problem at all! Even TokuMX clearly says on their internet site that its no big deal for TokuMX if data + index does not fit in RAM.

Also, for MongoDB respective for TokuMX we can limit the RAM usage by some commands.

For my web project I would like to decide which database I am going to use, I dont want to change later. The RAM of my database server is not more than 500MB right now, and it is used concurrenctly by the NodeJS server. So both sit on one server.

On that server I have 1 Nodejs Server and 2 Database instances running. Hereby I compare TokuMX and ArangoDb with the TOP command in linux to check RAM usage. Both databeses just have a tiny collection for testing. And by the TOP command in Linux is says to me: ArangoDB: RES: 128 MByte in use, and for TokuMX it says only 9 MB (!!) Res means: actual, really used physical RAM I found out. So, the difference is already huge... thanks and many greetings. Also the virtual ram usage is enormously different. 5 Gb for arangodb. and just 300 MB for tokumx.

Answer 1

ArangoDB is a database plus an API sever. It uses V8 toifif extend the functionality of the database and define new APIs. In its default configuration on my Mac, it is using 3 scheduler threads and 5 V8 threads. This gives a resident size 81M. Setting the scheduler to 1 (which is normally enough) reduces this to 80M. Setting the V8 instances to two

./bin/arangod -c etc/relative/arangod.conf testbase3 --scheduler.threads 1 --server.threads 2

reduces this to 34M, setting it to 1 reduces it to 24M. Therefore presumably the database itself uses less than 14M.

Any mostly memory database (be it mongodb or arangodb or redis) will have a decreased performance, if the data and index grow too large. It might be OK, if your index still fits into memory and your working set of data is small enough. But if your working set becomes too large, your server will begin to swap and performance will decrease. This problem will exists in any type of mostly memory database like ArangoDB and MongoDB.

One additional question: Is it possible to use two different databases on one ArangoDB instance for you? It would be more memory efficient, to start just one server with two databases in it.

Answer 2

hello and thanks for your answer. Well, I will use one ArangoDB instance on my back server with Node instance only, as you recommended. No mongodb there anymore. I am sure I will have some questions in the future again. So far, I have to say, Arangosh is not that easy to hendle like mongo-shell. Its shell commands are a little bit cryptic to me. but the ACID transactions and the ability to run javascript on server side for production is a very big plus and really cool! And actually thats the reason why I use ArangoDB. Right now, from my Nodejs server I start the ACID transactions, which then sends the action and parameters to ArangoDb. And to minimize the action block I created a javascript module and I put it on ArangoDb in the directory you told me last time, and thats great. That little selfprogrammed javascript module does all the ACID transactions. But, do you agree, I probably could increase performance even more by programming an FOXX app for doing ACID transactions? Because right now I think everytime I call that selfmade module it needs to get loaded first bevor it can perform the transactions. And the foxx applications stays forever in RAM, and does not get reloaded for every hit. Do you agree?