Groovy neo4j batch import

0
votes

I am using this script http://jexp.de/blog/2014/10/flexible-neo4j-batch-import-with-groovy/ to import csv file to neo4j. The file is quite large: around 60 million lines. I allocated 30GB memory for properties, 30GB for nodes and 30GB for the rest. However, the importer gets stuck at 49 million lines. Is this the memory issue?

The output looks like this

48800000 lines processed in x seconds 48900000 lines processed in y seconds and then it stops processing and is not moving.

1
groovyneo4j
what is "stuck"? error, exception, nothing happens, 100% cpu, swapping, ...? – cfrick
nothing happens. I will update the question now – Anastasia
is your db server doing all the work then? flushing all batch informations or GCing the hell out of it? – cfrick
I don't understand your question. I am not a database person. I am using the code provided on github and adapted it to my needs in terms of properties and node indexing. Is there something else I should do? – Anastasia

1 Answers

1
votes

Did you miss relationships? Those are the most crucial And I forgot one in the original example, which is the

neostore.relgroupstore.db=10MB

15 bytes per node 38 bytes per relationship 42 bytes per 4 properties

And do you have actually 128GB of RAM? To accomodate for the memory mapping?

What OS are you using?

Oh and at the end it stops for quite a long time to create the indexes and constraints, that might be the case.

You can add an output before that.