Neo4j Unmanaged Extension and GuardTimeoutException

3
votes

I am in great need of some advice regarding an issue I'm running into with a Neo4j unmanaged extension that I'm building in Java. I have created a very simple code sample that highlights my issue. The basic premise is that I'd like to set the

org.neo4j.server.webserver.limit.executiontime

for the neo4j server to a reasonable amount of time for user queries (lets say 2 minutes) which are coming in through Cypher, other endpoints, etc. But I also have some batch jobs that I need to run through this unmanaged extension, and therefore I attempted to break them up into several <2 minute transactions. The issue I am seeing is that even though each of my transactions is <2 minutes, once my process has been running for 2 minutes I get a GuardTimeoutException.

Here is the sample. In this sample note that I have limited the time to 2000 milliseconds, so it doesn't take me all day to debug. (getting awful close though!)

Endpoint

/**
 * Sample endpoint.
@GET
@Path("test")
@Produces(MediaType.APPLICATION_JSON)
public Response test(@Context GraphDatabaseService service) {
    TestFile.test(service);
    return Response.ok().build();
}

Logic

public static void test(final GraphDatabaseService service) {
    for (int i = 0; i < 100000; i++) {
        try (Transaction tx = service.beginTx();) {
            final Node n = service.createNode();
            n.addLabel(testLabel);
            tx.success();
            tx.close();
        }
        System.out.println("Added node");
    }
}

I can see that each transaction takes only a fraction of a second, since I load in about 200 nodes successfully before timeout. At exactly 2 seconds from hitting the endpoint though, I get the following:

org.neo4j.kernel.guard.GuardTimeoutException: timeout occured (overtime=1)
at org.neo4j.kernel.guard.Guard$Timeout.check(Guard.java:132)
at org.neo4j.kernel.guard.Guard.check(Guard.java:43)
at org.neo4j.kernel.InternalAbstractGraphDatabase$5.createNode(InternalAbstractGraphDatabase.java:794)
at org.neo4j.kernel.impl.api.state.OldTxStateBridgeImpl.nodeCreate(OldTxStateBridgeImpl.java:120)
at org.neo4j.kernel.impl.api.state.TxStateImpl.nodeDoCreate(TxStateImpl.java:366)
at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.nodeCreate(StateHandlingStatementOperations.java:99)
at org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.nodeCreate(ConstraintEnforcingEntityOperations.java:390)
at org.neo4j.kernel.impl.api.LockingStatementOperations.nodeCreate(LockingStatementOperations.java:207)
at org.neo4j.kernel.impl.api.OperationsFacade.nodeCreate(OperationsFacade.java:506)
at org.neo4j.kernel.InternalAbstractGraphDatabase.createNode(InternalAbstractGraphDatabase.java:1120)
at **TestFile.test(TestFile.java:15)
at **test(Jobs.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at org.neo4j.server.rest.transactional.TransactionalRequestDispatcher.dispatch(TransactionalRequestDispatcher.java:139)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506)
at org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
at org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:445)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:744)

As you can see I have tried tx.success(), tx.close(), anything I can think of to make all the transactions die when they are completed. Any advice greatly appreciated!!

update -------------

Michael - I have followed the steps that you suggested - I have a new java class that extends SPIPluginLifecycle and have also added the new config file in /src/main/resources/META-INF/services. I can see that this file ends up in the jar that goes in /plugins within META-INF/services. However, I don't see this initialization class getting called at all. Heres the class and logs I do see on startup. 

public class GraphInitializer implements SPIPluginLifecycle {
private WebServer webServer;

@Override
public Collection<Injectable<?>> start(final GraphDatabaseService graphDatabaseService, final Configuration config) {
    throw new IllegalAccessError();
}

@Override
public void stop() {

}

@Override
public Collection<Injectable<?>> start(final NeoServer neoServer) {
    System.out.println("K starting!");
    webServer = getWebServer(neoServer);
    final Database database = neoServer.getDatabase();
    final GraphDatabaseAPI graphDatabaseAPI = database.getGraph();
    final Guard guard = graphDatabaseAPI.getDependencyResolver().resolveDependency(Guard.class);
    final Filter filter = new GuardingRequestFilter(guard, 3000);
    webServer.addFilter(filter, "/*" );
    return null;
}

private WebServer getWebServer(final NeoServer neoServer) {
    if (neoServer instanceof AbstractNeoServer) {
        return ((AbstractNeoServer) neoServer).getWebServer();
    }
    throw new IllegalArgumentException("expected AbstractNeoServer");
}

Logs:

2014-10-06 16:14:23.009+0000 INFO  [API] Mounted discovery module at [/]
2014-10-06 16:14:23.014+0000 INFO  [API] Mounted REST API at [/db/data/]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted management API at [/db/manage/]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted third-party JAX-RS package [***] at [/kristen]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted webadmin at [/webadmin]
2014-10-06 16:14:23.017+0000 INFO  [API] Mounted Neo4j Browser at [/browser]
2014-10-06 16:14:23.070+0000 INFO  [API] Mounting static content at [/webadmin] from [webadmin-html]
2014-10-06 16:14:23.124+0000 INFO  [API] Mounting static content at [/browser] from [browser]
12:14:23.126 [main] WARN  o.e.j.server.handler.ContextHandler - o.e.j.s.ServletContextHandler@1cb1c025{/,null,null} contextPath ends with /
12:14:23.127 [main] WARN  o.e.j.server.handler.ContextHandler - Empty contextPath
12:14:23.131 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.0.5.v20130815
12:14:23.155 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.h.MovedContextHandler@5de15910{/,null,AVAILABLE}
12:14:23.245 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /webadmin, did not find org.apache.jasper.servlet.JspServlet
12:14:23.255 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@4b45c74e{/webadmin,jar:file:/usr/local/Cellar/neo4j/2.1.3/libexec/system/lib/neo4j-server-2.1.3-static-web.jar!/webadmin-html,AVAILABLE}
12:14:23.668 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@2917eb65{/kristen,null,AVAILABLE}
12:14:23.817 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@54589209{/db/manage,null,AVAILABLE}
12:14:24.003 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@68a1f8e7{/db/data,null,AVAILABLE}
12:14:24.021 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /browser, did not find org.apache.jasper.servlet.JspServlet
12:14:24.022 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@438e0daa{/browser,jar:file:/usr/local/Cellar/neo4j/2.1.3/libexec/system/lib/neo4j-browser-2.1.3.jar!/browser,AVAILABLE}
12:14:24.103 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@1cb1c025{/,null,AVAILABLE}
12:14:24.115 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@75b61dd7{HTTP/1.1}{localhost:7474}
12:14:24.503 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@6464d61f{SSL-HTTP/1.1}{localhost:7473}
2014-10-06 16:14:24.503+0000 INFO  [API] Server started on: http://localhost:7474/
2014-10-06 16:14:24.504+0000 INFO  [API] Remote interface ready and available at [http://localhost:7474/]

I expected a new line entry or something - also my changes to add a timeout don't actually work so I'm sure these changes haven't taken affect. Is there anything additional I have to add to the neo4j-server.properties, neo4j.properties, etc? I have already successfully added the line that sets up the unmanaged extension.

thank you!

2
javaneo4jjerseycyphergraph-databases

2 Answers

1
votes

Small transaction make it go slower as each has to be flushed to disk. Usually it makes sense to add up to 50k nodes in one tx, which should take less than a second.

For your guard filter, it's just a servlet filter, so you can also just check the Neo4j source code where it is installed and on init of your extension install it yourself but excempt the endpoints you want to batch to.

In: org.neo4j.server.AbstractNeoServer line 488

    Filter filter = new GuardingRequestFilter( guard,
            getConfiguration().getInt( Configurator.WEBSERVER_LIMIT_EXECUTION_TIME_PROPERTY_KEY ) );
    webServer.addFilter( filter, "/*" );

You can use the extension initializer to add your custom filter like shown here:

https://github.com/neo4j-contrib/authentication-extension/blob/2.1/src/main/java/org/neo4j/server/extension/auth/AuthenticationExtensionInitializer.java#L81

you have to add a file like this which contains your initializer class name:

https://github.com/neo4j-contrib/authentication-extension/blob/2.1/src/main/resources/META-INF/services/org.neo4j.server.plugins.PluginLifecycle

0
votes

Solution Posted by Michael Hunger worked perfectly, with a few of my own learning points along the way.

  1. I had to update the org.neo4j.server.thirdparty_jaxrs_classes property in neo4j-server.properties to a more root directory, since I had not put my initializer in the same directory as my api java files.
  2. The start() method had to return an empty collection.
  3. I was able to use several different filters to specify time limits for different endpoints.

Final code below:

@Override
public Collection<Injectable<?>> start(final NeoServer neoServer) {
    webServer = getWebServer(neoServer);
    final Database database = neoServer.getDatabase();
    final GraphDatabaseAPI graphDatabaseAPI = database.getGraph();
    final Guard guard = graphDatabaseAPI.getDependencyResolver().resolveDependency(Guard.class);
    final Filter filter = new GuardingRequestFilter(guard, 600);
    webServer.addFilter(filter, "/*" );
    final Filter filter2 = new GuardingRequestFilter(guard, 3000);
    webServer.addFilter(filter2, "*/batch" );
    return Collections.emptyList();
}