Neo4j outrageous write performance

0
votes

I'm doing performance tests for neo4j 3.0.7 using the new bolt client and I see some strange results. I'm trying a very simple scenario: upserting 1000 nodes (using merge command).

I tried several approaches:

  1. 1000 synchronous transactions of 1 command
  2. 1 transaction of 1000 command
  3. 1000 asynchronous transactions of 1 command (using 10 threads)

This is the query I am executing (I have a uniqueness constraint on person.id): "Merge (n:person {id:'123'}) SET n.name='Diana Kesha', n.address='aaa' .... RETURN n.id"

Here is the execution plan of my query:

Execution plan

Here are the results:

  1. ~3sec per 1000 synchronous transactions (notice it not 1000k)
  2. ~2sec per 1 transaction of 1000 commands
  3. ~2.5sec per 1000 asynchronous transactions

Absolutely unacceptable results for a 64 cores with 128G of RAM machine!

Now, going deeper, I noticed that Neo4j is using 25% of RAM (which is fine I think) but only 1 core when using the 1st 2 options, and 10 cores when using 10 threads.

also I noticed that changing Neo4j's working threads has no impcat on the amount of actualy cores being used.

What am I missing ?

1
javaneo4j
What queries are you executing? How are you executing them? What indexes do you have? - Ant P
Also, have you run PROFILE with your queries? For queries with lots of db hits, you may want to paste that into your description, expanding all nodes of the profile graph. - InverseFalcon
I added the query and the explanation plain. But still, it doesn't explain the fact that only one core is participating in the process - Ido Nadler
Please leave the image inline, it's really easier to read in context than going to another tab. - Frank Pavageau
Are you using the exact query in your description for every query, MERGING the same node each time, or is the data different and unique for all merges? - InverseFalcon

1 Answers

2
votes

You have 2 conflicting numbers, do you mean 1000 or 1000k = 1M ?

2 s for 1M updates is too slow ?

I would recommend doing 1k to 10k updates per tx and then parallelize the tx

  1. You are not using parameters
  2. don't return anything
  3. use capitalized labels (adapt the constraint)

like:

MERGE (n:Person {id:{id}}) 
 ON CREATE SET n.name={name} n.address={address}

better to batch the (2) into a list parameter

UNWIND {data} as row 
MERGE (n:Person {id:row.id}) 
ON CREATE SET n.name=row.name n.address=row.address

So test 1000 requests (parallel) with 1000 updates each, sending a list with 1000 maps each time.

UNWIND {data} as row 
MERGE (n:Person {id:row.id}) 
ON CREATE SET n.name=row.name n.address=row.address