Similar Gremlin Queries Written in Different Ways Have Performance Differences

Question

I have two gremlin traversal queries pretty much doing the same thing, each looks up a vertex with ID and upserts a new node with an edge to the new node.

I tried with this first because this looked cleaner:

g.V('22b515e0-dbefb359-10f3-71ff13527bb2').sideEffect(__.outE().drop())\
.addE('someedge').to(__.V().hasLabel('SomeLabel').has('name', \
'SomeName').sideEffect(__.properties().drop()).property('name123', \
'SomeName123')).next()

This query takes around 8 seconds every time it executes.

As a measure to fix the above issue, I tried changing the traversal to this and the response time is in milliseconds:

g.V().hasLabel('SomeLabel').has('name', \
'SomeName').sideEffect(__.properties().drop()).property('name123', \
'SomeName123').as_('X').V('22b515e0-dbef-b359-10f3-71ff13527bb2')\
.sideEffect(__.outE().drop()).addE('some \
edge').to(__.select('X')).next()

But, when I lookup using ID directly (In method - 1), even with this the response time is in milliseconds.

g.V('22b515e0-dbef-b359-10f3-71ff13527bb2').sideEffect(__.outE().drop())\
.addE('someedge').to(__.V('8b66cab2-3b1c-41c9-9acb-296dda3c9139')\
.sideEffect(__.properties().drop()).property('name123', \
'SomeName123')).next()

Here are some .profile() metrics which shows execution times. For 1:

{'dur': 8823.148244,
 'metrics': [{'dur': 0.336926,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NeptuneGraphQueryStep(Vertex)',
   'annotations': {'percentDur': 0.003818659628994895},
   'id': '6.0.0()'},
  {'dur': 1.882282,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'TraversalSideEffectStep([VertexStep(OUT,edge), ProfileStep, DropStep, ProfileStep])',
   'annotations': {'percentDur': 0.02133345091736396},
   'id': '1.0.0()',
   'metrics': [{'dur': 0.275912,
     'counts': {'traverserCount': 11, 'elementCount': 11},
     'name': 'VertexStep(OUT,edge)',
     'id': '0.1.0(1.0.0())'},
    {'dur': 1.569401,
     'counts': {},
     'name': 'DropStep',
     'id': '1.1.0(1.0.0())'}]},
  {'dur': 0.115423,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NoOpBarrierStep',
   'annotations': {'percentDur': 0.0013081838455847213},
   'id': '5.0.0()'},
  {'dur': 8820.813613,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'AddEdgeStep({~to=[[GraphStep(vertex,[]), ProfileStep, NeptuneHasStep([~label.eq(SomeLabel), name.eq(SomeName)]), ProfileStep, TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep]), ProfileStep, NoOpBarrierStep, ProfileStep, AddPropertyStep({value=[SomeName123], key=[name123]}), ProfileStep]], label=[some edge]})',
   'annotations': {'percentDur': 99.97353970560806},
   'id': '2.0.0()',
   'metrics': [{'dur': 1346.058324,
     'counts': {'traverserCount': 373732, 'elementCount': 373732},
     'name': 'GraphStep(vertex,[])',
     'id': '0.1.0(2.0.0())'},
    {'dur': 7473.661772,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'NeptuneHasStep([~label.eq(SomeLabel), name.eq(SomeName)])',
     'id': '5.1.0(2.0.0())'},
    {'dur': 0.283139,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep])',
     'id': '2.1.0(2.0.0())',
     'metrics': [{'dur': 0.096149,
       'counts': {'traverserCount': 1, 'elementCount': 1},
       'name': 'PropertiesStep(property)',
       'id': '0.2.0(2.1.0(2.0.0()))'},
      {'dur': 0.149241,
       'counts': {},
       'name': 'DropStep',
       'id': '1.2.0(2.1.0(2.0.0()))'}]},
    {'dur': 0.046801,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'NoOpBarrierStep',
     'id': '4.1.0(2.0.0())'},
    {'dur': 0.306436,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'AddPropertyStep({value=[SomeName123], key=[name123]})',
     'id': '3.1.0(2.0.0())'}]}]}

For 2:

{'dur': 5.596631,
 'metrics': [{'dur': 0.510122,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NeptuneGraphQueryStep(Vertex)',
   'annotations': {'percentDur': 9.114804960341319},
   'id': '11.0.0()'},
  {'dur': 0.462721,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep])',
   'annotations': {'percentDur': 8.267848997012667},
   'id': '2.0.0()',
   'metrics': [{'dur': 0.179074,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'PropertiesStep(property)',
     'id': '0.1.0(2.0.0())'},
    {'dur': 0.210574,
     'counts': {},
     'name': 'DropStep',
     'id': '1.1.0(2.0.0())'}]},
  {'dur': 0.092257,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NoOpBarrierStep',
   'annotations': {'percentDur': 1.6484381407314508},
   'id': '9.0.0()'},
  {'dur': 1.653211,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'AddPropertyStep({value=[SomeName123], key=[name123]})@[X]',
   'annotations': {'percentDur': 29.539396111696483},
   'id': '3.0.0()'},
  {'dur': 0.116915,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'GraphStep(vertex,[22b515e0-dbef-b359-10f3-71ff13527bb2])',
   'annotations': {'percentDur': 2.0890246292814374},
   'id': '4.0.0()'},
  {'dur': 1.073502,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'TraversalSideEffectStep([VertexStep(OUT,edge), ProfileStep, DropStep, ProfileStep])',
   'annotations': {'percentDur': 19.181218129263836},
   'id': '5.0.0()',
   'metrics': [{'dur': 0.14199,
     'counts': {'traverserCount': 11, 'elementCount': 11},
     'name': 'VertexStep(OUT,edge)',
     'id': '0.1.0(5.0.0())'},
    {'dur': 0.879906,
     'counts': {},
     'name': 'DropStep',
     'id': '1.1.0(5.0.0())'}]},
  {'dur': 0.042369,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NoOpBarrierStep',
   'annotations': {'percentDur': 0.7570447292308533},
   'id': '10.0.0()'},
  {'dur': 1.645534,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'AddEdgeStep({~to=[[SelectOneStep(last,X), ProfileStep]], label=[some edge]})',
   'annotations': {'percentDur': 29.40222430244195},
   'id': '6.0.0()',
   'metrics': [{'dur': 0.024292,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'SelectOneStep(last,X)',
     'id': '0.1.0(6.0.0())'}]}]}

For 3:

{'dur': 9.084957,
 'metrics': [{'dur': 0.340621,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NeptuneGraphQueryStep(Vertex)',
   'annotations': {'percentDur': 3.7492857698721083},
   'id': '6.0.0()'},
  {'dur': 1.92865,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'TraversalSideEffectStep([VertexStep(OUT,edge), ProfileStep, DropStep, ProfileStep])',
   'annotations': {'percentDur': 21.229049295445208},
   'id': '1.0.0()',
   'metrics': [{'dur': 0.282423,
     'counts': {'traverserCount': 11, 'elementCount': 11},
     'name': 'VertexStep(OUT,edge)',
     'id': '0.1.0(1.0.0())'},
    {'dur': 1.578817,
     'counts': {},
     'name': 'DropStep',
     'id': '1.1.0(1.0.0())'}]},
  {'dur': 0.058816,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'NoOpBarrierStep',
   'annotations': {'percentDur': 0.647399872118272},
   'id': '5.0.0()'},
  {'dur': 6.75687,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'AddEdgeStep({~to=[[GraphStep(vertex,[8b66cab2-3b1c-41c9-9acb-296dda3c9139]), ProfileStep, TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep]), ProfileStep, NoOpBarrierStep, ProfileStep, AddPropertyStep({value=[SomeName123], key=[name123]}), ProfileStep]], label=[some edge]})',
   'annotations': {'percentDur': 74.37426506256442},
   'id': '2.0.0()',
   'metrics': [{'dur': 0.087179,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'GraphStep(vertex,[8b66cab2-3b1c-41c9-9acb-296dda3c9139])',
     'id': '0.1.0(2.0.0())'},
    {'dur': 0.332149,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep])',
     'id': '1.1.0(2.0.0())',
     'metrics': [{'dur': 0.144028,
       'counts': {'traverserCount': 1, 'elementCount': 1},
       'name': 'PropertiesStep(property)',
       'id': '0.2.0(1.1.0(2.0.0()))'},
      {'dur': 0.159151,
       'counts': {},
       'name': 'DropStep',
       'id': '1.2.0(1.1.0(2.0.0()))'}]},
    {'dur': 0.052224,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'NoOpBarrierStep',
     'id': '3.1.0(2.0.0())'},
    {'dur': 5.781101,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'AddPropertyStep({value=[SomeName123], key=[name123]})',
     'id': '2.1.0(2.0.0())'}]}]}

There are around 300k vertices and 3 million edges in the graph database (AWS Neptune).

Am I missing something here? I wanted to make it work with method-1 as this is a part of a complex traversal which might not work well with method-2

stephen mallette stephen mallette · Accepted Answer · 2019-04-30T18:39:45

Judging from the profile() of the three queries, Neptune is not optimizing the child traversal of to(V().hasLabel('SomeLabel').has('name','SomeName')). It's doing a full graph scan for V() in that case denoted by this:

{'dur': 8820.813613,
   'counts': {'traverserCount': 1, 'elementCount': 1},
   'name': 'AddEdgeStep({~to=[[GraphStep(vertex,[]), ProfileStep, NeptuneHasStep([~label.eq(SomeLabel), name.eq(SomeName)]), ProfileStep, TraversalSideEffectStep([PropertiesStep(property), ProfileStep, DropStep, ProfileStep]), ProfileStep, NoOpBarrierStep, ProfileStep, AddPropertyStep({value=[SomeName123], key=
[name123]}), ProfileStep]], label=[some edge]})',
   'annotations': {'percentDur': 99.97353970560806},
   'id': '2.0.0()',
   'metrics': [{'dur': 1346.058324,
     'counts': {'traverserCount': 373732, 'elementCount': 373732},
     'name': 'GraphStep(vertex,[])',
     'id': '0.1.0(2.0.0())'},
    {'dur': 7473.661772,
     'counts': {'traverserCount': 1, 'elementCount': 1},
     'name': 'NeptuneHasStep([~label.eq(SomeLabel), name.eq(SomeName)])',
     'id': '5.1.0(2.0.0())'},

See the traverser count for V() is 373732 which feeds into has() which does a 7 second in-memory filter to find 1 vertex.

Similar Gremlin Queries Written in Different Ways Have Performance Differences

1 Answers