Load a large Collection on ArangoDB

1
votes

I'm working on a project using ArangoDB. This project implies large data (not too much at now) as 4000k documents in a single Collection.

I migrated the data step by step and at the end there was no problem on reading Collection data.

We stopped ArangoDB and our machine because weekend. When we started our machine it was impossible to load data of that Collection.

Sequentially we tried to:

  • Load data by arangodb client, but response was been a generic error 'Collection error'
  • Dump data from console: but response was been: Error: got invalid response from server: Could not connect to 'tcp://localhost:8529' 'connect() failed with #111 - Connection refused'

What can we do? This collection has a good number of indexes and the bucket was fixed to 64.

EDIT I tried to stop and start again arangodb: problem persist. So, I would excluding that is a server problem. Could be possible that there is some wrong configuration in Collection (wait for sync: false, bucket 64, journal size: 32)?

1
javascriptdatabaseindexingarangodb
What version are you using and what is your environment? Does the server process die when you try to connect and read data from that collection? BTW: It's probably better to post this as bug report: github.com/arangodb/arangodb/issues – CodeManX
ArangoDB: 2.8.7 OS: Ubuntu 14.04 Nodejs: 4.4.2 – carmelolg
can you check for relevant log messages in /var/log/arangodb/* ? – dothebart
I already checked. There are no relevant messages. @dothebart – carmelolg
can you upgrade to the latest 2.8 release, and check whether the error persists? – dothebart

1 Answers

0
votes

The original question was solved by updating to the latest version from that release series.

The new question is howto choose the buckets for indices.

Buckets are memory areas in which the indices are kept. Default is to have 8 such buckets. Once one bucket is full, a new bucket of double size is allocated, existing structures are copied over. Copying comes at a cost, which will slow down index build up, or insert of new items.

So if the collection you want to work with is big and / or has many inserts, a bigger number of buckets may work better in your case. You may want to double the bucket count and see whether it works better for you.