Neo4j performance - is it really fast?

3
votes

I'm wanting to run some tests on neo4j, and compare its performance with other databases, in this case postgresql.

This postgres database have about 2000000 'content's distributed around 3000 'categories'. ( this means that there is a table 'content', one 'category' and a relation table 'content-to-category' since one content can be in more than 1 category). So, mapping this to a neo4j db, i'm creating nodes 'content', 'category' and their relations ( content to category, and content to content, cause contents can have related contents).

category -> category ( categories can have sub-categories )

content -> category

content -> content (related)

Do you think this 'schema' is ok for this type of domain ?

  1. migrating all data from postgresql do neo4j: it is taking forever ( about 4, 5 days ). This is just some search for nodes and creating/updating accordingly. (search is using indexes and the insert/update if taking 500ms for each node) Am i doing something wrong ?

  2. Migration is done, so i went to try some querying ... i ended up with about 2000000 content nodes, 3000 category nodes, and more than 4000000 relationships.

(please note that i'm new to all this neo4j world, so i have no idea how to optimize cypher queries...)

One of the queries i wanted to test is: get the 10 latest published contents of a given 'definition' in a given category (this includes contents that are in sub categories of the given category)

experimenting a little, i ended up with something like this :

START 
  c = node : node_auto_index( 'type: category AND code: category_code' ),
  n = node : node_auto_index( 'type: content AND state: published AND definitionCode: definition_name' )
MATCH (c) <- [ r:BELONGS_TO * ] - (n)
RETURN n.published_stamp, n.title
ORDER BY n.published_stamp DESC
LIMIT 6

this takes around 3 seconds, excluding the first run, that takes a lot more ... is this normal ?

What am i doing wrong ?

please note that i'm using neo4j 1.9.2, and auto indexing some node properties ( type, code, state, definitionCode and published_stamp included - title is not auto indexed )

also, returning 'c' on the previous query ( start c = node : node_auto_index( 'type: category AND code : category-code' ) return c; ) is fast (again, excluding the first run, that takes aroung 20-30ms) also, i'm not sure if this is the right way to use indexes ...

Thank you in advance (sorry if something is not making sense - ask me and i'll try to explain better).

3
neo4j

3 Answers

2
votes

Have you looked at the batch import facilities: http://www.neo4j.org/develop/import? You really should look at that for the initial import - it will take minutes instead of days.

I will ask some of our technical folks to get back to you on some of the other stuff. You really should not be seeing this.

Rik

1
votes

How many nodes are returned by this?

START 
  n = node : node_auto_index( 'type: content AND state: published AND definitionCode: definition_name' )
RETURN count(*)

I would try to let the graph do the work. How deep are your hierarchies usually? Usually you limit arbitrary length relationships to not have the combinatorial explosion: I would also have a different relationship-type between content and category than the category tree. Can you point out your current relationship-types?

START 
  c = node : node_auto_index( 'type: category AND code: category_code' ),
MATCH (c) <- [:BELONGS_TO*5] - (n)
WHERE n.type = 'content' AND n.state='published' and n.definitionCode = 'definition_name'
RETURN n.published_stamp, n.title
ORDER BY n.published_stamp DESC
LIMIT 6

Can you try that?

For import it is easiest to generate CSV from your SQL and import that using http://github.com/jexp/batch-import

0
votes

Are you running Linux, maybe on an ext4 filesystem?

You might want to set the barrier=0 mount option, as described here: http://structr.org/blog/neo4j-performance-on-ext4

Further discussion of this topic: https://groups.google.com/forum/#!topic/neo4j/nflUyBsRKyY