Splitting Big Model in Neo4j

1
votes

I have a huge amount of data and want to create a model in Neo4j representing this data.

It would be about 3 million nodes and more than 3 billion relationships. Building this with the Batch Inserter takes too long to import the data and then create the nodes and relationships.

The question is: can I split the huge model into two separate models and then run a cypher query that accesses the two models at the same time? If yes, how do I do it?

1
neo4jdata-modelingcypherbatch-insert
The batch-inserter can insert 1M nodes/second and 500k rels/second if you give it enough memory for memory mapping the files. So that shouldn't be such a big issue. - Michael Hunger
How did you try to use the batch-inserter? - Michael Hunger
I have to get the data from Oracle DB, and then loop over the returned resultset, and create the nodes.. After that I have another result set representing the connected ids that will have relationships - Mohamed E. ManSour
Mohamed, I've been playing around with doing this in a batch inserter fashion, one thing to note on speed, set your rowFetchSize to be something big. By default it's 10, and my Data load of a billion records took 13 hours. I set it to 10,000,000 and it now takes 20 minutes. - Nicholas

1 Answers

2
votes

I'm afraid Neo4j does not natively support partitioning the graph across multiple instances of the database (yet).

However, 3M nodes and 3B edges isn't a huge amount of data. What exactly do you mean by "too long"?