Intermittent EntityNotFoundException with Spring Data Neo4j

Question

I'm using Neo4j 2.1.6 and Spring Data Neo4j 3.2.1. I've been using Neo4j 2.1.6 the whole time (i.e. haven't migrated from an earlier version of Neo).

I'm running into an intermittent issue where when I try to do a MATCH statement, e.g.

MATCH (p:Profile) WHERE p.username = 'abc123' RETURN p

I'm getting an EntityNotFoundException, saying things like,

Node with id 25393

A couple similar MATCH queries return similar results.

I'm noticing that the ids that aren't found are greater than the max internal node ID in the database.

I'm not exactly sure what's going on as it's intermittent.

Any help on the matter would be greatly appreciated.

Example stack trace follows:

</pre><p><b>root cause</b></p><pre>org.neo4j.graphdb.NotFoundException: Node 25393 not found
org.neo4j.kernel.impl.core.NodeManager.getNodeById(NodeManager.java:323)
org.neo4j.kernel.InternalAbstractGraphDatabase.getNodeById(InternalAbstractGraphDatabase.java:1177)
org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$NodeOperations.getById(TransactionBoundQueryContext.scala:161)
org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:135)
org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:135)
org.neo4j.cypher.internal.helpers.JavaConversionSupport$$anon$2.next(JavaConversionSupport.scala:33)
scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
scala.collection.Iterator$$anon$13.next(Iterator.scala:372)
scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
org.neo4j.cypher.internal.compiler.v2_1.pipes.EagerAggregationPipe.internalCreateResults(EagerAggregationPipe.scala:78)
org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:105)
org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)
org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)
org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:102)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:120)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:119)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionWorkflowBuilder.runWithQueryState(ExecutionPlanBuilder.scala:168)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:103)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:68)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:67)
org.neo4j.cypher.internal.ExecutionPlanWrapperForV2_1.execute(CypherCompiler.scala:159)
org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:76)
org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:71)
org.neo4j.cypher.javacompat.ExecutionEngine.execute(ExecutionEngine.java:84)
org.springframework.data.neo4j.support.query.CypherQueryEngineImpl.parseAndExecuteQuery(CypherQueryEngineImpl.java:70)
org.springframework.data.neo4j.support.query.CypherQueryEngineImpl.query(CypherQueryEngineImpl.java:58)
org.springframework.data.neo4j.repository.query.GraphRepositoryQuery.dispatchQuery(GraphRepositoryQuery.java:107)
org.springframework.data.neo4j.repository.query.GraphRepositoryQuery$1.doWithGraph(GraphRepositoryQuery.java:89)
org.springframework.data.neo4j.support.Neo4jTemplate.doExecute(Neo4jTemplate.java:457)
org.springframework.data.neo4j.support.Neo4jTemplate.access$000(Neo4jTemplate.java:87)
org.springframework.data.neo4j.support.Neo4jTemplate$2.doInTransaction(Neo4jTemplate.java:471)
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
org.springframework.data.neo4j.support.Neo4jTemplate.exec(Neo4jTemplate.java:468)
org.springframework.data.neo4j.repository.query.GraphRepositoryQuery.execute(GraphRepositoryQuery.java:83)
org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:421)
org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:381)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
com.sun.proxy.$Proxy78.usernameExists(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
com.sun.proxy.$Proxy79.usernameExists(Unknown Source)
com.theapp.dao.impl.UserDAOImpl.usernameExists(UserDAOImpl.java:181)
com.theapp.service.impl.RegistrationServiceImpl.registerUserAccount(RegistrationServiceImpl.java:45)
com.theapp.facade.impl.UserFacadeImpl.registerUser(UserFacadeImpl.java:137)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
com.sun.proxy.$Proxy171.registerUser(Unknown Source)
com.theapp.controllers.AuthController.registration(AuthController.java:57)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:781)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:721)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
com.theapp.security.CustomAuthenticationProcessingFilter.doFilter(CustomAuthenticationProcessingFilter.java:101)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)

Answer 1

It seems you have no index on :Profile(username)

So Neo (Cypher) has to scan over all :Profile nodes some of which might have been removed concurrently. So Cypher reads the id from the index, after that the node is deleted and only later it tries to load the node with that id for property comparison but it already has gone.

I think it's addressed in a more current version of Neo4j but I'm not sure.

Answer 2

We ran into a similar problem on Neo4j 2.2.3 and Spring Data 3.3.1.

Based on your comment to Michael's answer, it sounds like you have an index on the username property. We verified that we have a schema index against "username" using :schema in the Neo4j browser. It seems the index is somehow corrupted. I verified this by running the same query but with a couple of twists to bypass the index:

MATCH (p:Profile) USING SCAN p:Profile WHERE p.username = 'abc123' RETURN p
MATCH (p:Profile) WITH p WHERE p.username = 'abc123' RETURN p

The above queries would work, but the original query you gave failed as described. Sometimes queries would fail by returning no results (when they should have results), and sometimes we saw some Neo4j error about a missing ID. This means the queries only work when not hitting the index.

Then running

DROP INDEX ON :Profile(username)
CREATE INDEX ON :Profile(username)

seemed to fix it, confirming that the index is corrupt. However, the whole time Neo4j claimed the username index was ONLINE. This happened multiple times, but we have no steps to reproduce.

This is likely a bug in Neo4j. My recommendation is to avoid (drop all) schema property indexes until it is fixed.