How to do efficient bulk upserts (insert new vertex, or update property(ies)) in Gremlin?

Question

Context:

I do have a graph with about 2000 vertices, and 6000 edges, this over time might grow to 10000 vertices and 100000 edges. Currently I am upserting the new vertices using the following traversal query:

Upserting Vertices & Edges

queryVertex = "g.V().has(label, name, foo).fold().coalesce(
               unfold(), addV(label).property(name, foo).property(model, 2)
               ).property(model, 2)"

The intent here is to look for vertex, named foo, and if found update its model property, otherwise create a new vertex and set the model property. this is issued twice: once for the source vertex and then for the target vertex.
Once the two related vertices are created, another query is issued to create the edge between them:

queryEdge = "g.V('id_of_source_vertex').coalesce(
             outE(edge_label).filter(inV().hasId('id_of_target_vertex')), 
             addE(edge_label).to(V('id_of_target_vertex'))
             ).property(model, 2)"

here, if there is an edge between the two vertices, the model property on edge is updated, otherwise it creates the edge between them.

And the pseudocode that does this, is something as follows:

for each edge in the list of new edges:
   //upsert source and target vertices:  
   execute queryVertex for edge.source
   execute queryVertex for edge.target
   // upsert edge: 
   execute queryEdge

This works, but it is highly inefficient; for example for the mentioned graph size it takes several minutes to finish, and with some in-app concurrency, it reduces the time only by couple of minutes. Surely, there must be a more efficient way of doing this for such a small graph size.

Question
* How can I make these upserts faster?

stephen mallette stephen mallette · Accepted Answer · 2020-06-11T17:46:40

Bulk loading should typically be relegated to the provider specific tools that are optimized to handle such tasks. Gremlin really doesn't provide abstractions to cover the diverse group of bulk loader tools that are out there for each of the various graph database systems that implement TinkerPop. For Neptune, which is how you tagged your question, that would mean using the Neptune Bulk Loader.

Speaking specifically to your question, though you might see some optimizations to what you described as your approach. From a Gremlin perspective, I imagine you would see some savings here by submitting a single Gremlin request per edge by combining your existing traversals:

g.V().has(label, name, foo).fold().
  coalesce(unfold(), 
           addV(label).property(name, foo)).
  property(model, 2).as('source').
  V().has(label, name, bar).fold().
  coalesce(unfold(), 
           addV(label).property(name, bar)).
  property(model, 2).as('target').
  coalesce(inE(edge_label).where(outV().as('source')), 
           addE(edge_label).from('source').to('target')).
  property(model, 2)

I think I got that right - untested, but hopefully you get the idea. Basically, we just reference the vertices already in memory via step labels so that we don't need to requery them. You might try other tactics as well if you continue with Gremlin-style bulk loading like ordering your edges so that you could batch together more edge loads to reduce the amount of vertex lookups and submit vertex/edge data in a more dynamic fashion as described here.

How to do efficient bulk upserts (insert new vertex, or update property(ies)) in Gremlin?

1 Answers