Embedded jetty processes requests running from eclipse, but not deployed

2
votes

I have an application embedding Jetty as the web server to host an RESTful API. I've been developing it in Eclipse on Ubuntu, and when launched from the IDE everything works great. When I package up the application, however, and deploy it on a server Jetty no longer responds to requests.

The server does open a socket on the port I specify, and turning on INFO logging for spring-mvc shows my Controller and it's methods being mapped appropriately. Requests successfully open a connection, but then no data is sent back and in fact the request never makes it to my controller code. When I shut down the application I see all of the connections that had been made get killed.

I've checked that all the libraries on the classpath for the Eclipse project are on the classpath when deployed. What else should I check to figure out why it work in Eclipse and is broken elsewhere?

Jetty is version 9, Java version 1.7, Spring 3.2.2.

EDIT Debug logging from Jetty when I make a request:

2013-05-06 09:40:30,214 DEBUG qtp1215430550-35-acceptor-0-ServerConnector@263676bd{HTTP/1.1}{0.0.0.0:8090} [SelectorManager.java submit]-: <Queued change org.eclipse.jetty.io.SelectorManager$ManagedSelector$Accept@1493f589>
2013-05-06 09:40:30,215 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java runChange]-: <Running change org.eclipse.jetty.io.SelectorManager$ManagedSelector$Accept@1493f589>
2013-05-06 09:40:30,217 DEBUG qtp1215430550-29-selector-2 [AbstractEndPoint.java onOpen]-: <onOpen SelectChannelEndPoint@38261f82{/127.0.0.1:39895<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@4edeac9a{false,null},wf=WriteFlusher@7d15dae9{IDLE},it=30000}{null}{io=0,kio=0,kro=0}>
2013-05-06 09:40:30,220 DEBUG qtp1215430550-29-selector-2 [IdleTimeout.java checkIdleTimeout]-: <SelectChannelEndPoint@38261f82{/127.0.0.1:39895<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@4edeac9a{false,null},wf=WriteFlusher@7d15dae9{IDLE},it=30000}{null}{io=0,kio=0,kro=0} idle timeout check, elapsed: 3 ms, remaining: 29997 ms>
2013-05-06 09:40:30,221 DEBUG qtp1215430550-29-selector-2 [HttpConnection.java <init>]-: <New HTTP Connection HttpConnection@5465f4d9{IDLE},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}>
2013-05-06 09:40:30,224 DEBUG qtp1215430550-29-selector-2 [AbstractConnection.java onOpen]-: <onOpen HttpConnection@5465f4d9{IDLE},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}>
2013-05-06 09:40:30,227 DEBUG qtp1215430550-29-selector-2 [AbstractConnection.java fillInterested]-: <fillInterested HttpConnection@5465f4d9{IDLE},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}>
2013-05-06 09:40:30,229 DEBUG qtp1215430550-29-selector-2 [SelectChannelEndPoint.java updateLocalInterests]-: <Local interests updated 0 -> 1 for SelectChannelEndPoint@38261f82{/127.0.0.1:39895<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@4edeac9a{true,AC.ExReadCB@5465f4d9},wf=WriteFlusher@7d15dae9{IDLE},it=30000}{HttpConnection@5465f4d9{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=1,kio=0,kro=0}>
2013-05-06 09:40:30,232 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java createEndPoint]-: <Created SelectChannelEndPoint@38261f82{/127.0.0.1:39895<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@4edeac9a{true,AC.ExReadCB@5465f4d9},wf=WriteFlusher@7d15dae9{IDLE},it=30000}{HttpConnection@5465f4d9{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=1,kio=0,kro=0}>
2013-05-06 09:40:30,233 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java runChange]-: <Running change org.eclipse.jetty.io.SelectChannelEndPoint$1@e56eabe>
2013-05-06 09:40:30,233 DEBUG qtp1215430550-29-selector-2 [SelectChannelEndPoint.java setKeyInterests]-: <Key interests updated 0 -> 1>
2013-05-06 09:40:30,234 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java select]-: <Selector loop waiting on select>
2013-05-06 09:40:30,234 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java select]-: <Selector loop woken up from select, 1/1 selected>
2013-05-06 09:40:30,234 DEBUG qtp1215430550-29-selector-2 [SelectChannelEndPoint.java setKeyInterests]-: <Key interests updated 1 -> 0>
2013-05-06 09:40:30,238 DEBUG qtp1215430550-29-selector-2 [SelectChannelEndPoint.java updateLocalInterests]-: <Local interests updated 1 -> 0 for SelectChannelEndPoint@38261f82{/127.0.0.1:39895<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@4edeac9a{true,AC.ExReadCB@5465f4d9},wf=WriteFlusher@7d15dae9{IDLE},it=30000}{HttpConnection@5465f4d9{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=0,kio=0,kro=1}>
2013-05-06 09:40:30,238 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java runChange]-: <Running change org.eclipse.jetty.io.SelectChannelEndPoint$1@e56eabe>
2013-05-06 09:40:30,239 DEBUG qtp1215430550-29-selector-2 [QueuedThreadPool.java dispatch]-: <qtp1215430550{STARTED,2<=10<=10,i=0,q=4} dispatched AC.ExReadCB@5465f4d9>
2013-05-06 09:40:30,240 DEBUG qtp1215430550-29-selector-2 [SelectorManager.java select]-: <Selector loop waiting on select>
2013-05-06 09:40:35,88 DEBUG Scheduler-702565976 [IdleTimeout.java checkIdleTimeout]-: <SelectChannelEndPoint@238b5d33{/127.0.0.1:39894<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@1293d180{false,null},wf=WriteFlusher@269f3407{IDLE},it=30000}{HttpConnection@1e10cb60{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=0,kio=0,kro=1} idle timeout check, elapsed: 29996 ms, remaining: 4 ms>
2013-05-06 09:40:35,96 DEBUG Scheduler-702565976 [IdleTimeout.java checkIdleTimeout]-: <SelectChannelEndPoint@238b5d33{/127.0.0.1:39894<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@1293d180{false,null},wf=WriteFlusher@269f3407{IDLE},it=30000}{HttpConnection@1e10cb60{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30005 ms, remaining: -5 ms>
2013-05-06 09:40:35,99 DEBUG Scheduler-702565976 [IdleTimeout.java checkIdleTimeout]-: <SelectChannelEndPoint@238b5d33{/127.0.0.1:39894<r-l>/127.0.0.1:8090,o=true,is=false,os=false,fi=FillInterest@1293d180{false,null},wf=WriteFlusher@269f3407{IDLE},it=30000}{HttpConnection@1e10cb60{INTERESTED},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of 0}}{io=0,kio=0,kro=1} idle timeout expired>

EDIT 2 My Jetty Server is configured all in spring beans:

<bean id="JettyServer" class="org.eclipse.jetty.server.Server" init-method="start" destroy-method="stop">

<property name="connectors">
    <list>
        <bean id="Connector" class="org.eclipse.jetty.server.nio.SelectChannelConnector">
            <property name="port" value="8090"/>
        </bean>
    </list>
</property>

<property name="handler">
    <bean id="handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">
        <property name="handlers">
            <list>
                <bean id="servletContextHandler" class="org.eclipse.jetty.servlet.ServletContextHandler">
                    <property name="contextPath" value="/"/>
                    <property name="servletHandler">
                        <bean class="org.eclipse.jetty.servlet.ServletHandler">

                            <property name="servlets">
                                <list>
                                    <bean class="org.eclipse.jetty.servlet.ServletHolder">
                                        <property name="name" value="DefaultServlet"/>
                                        <property name="servlet">
                                            <bean class="com.company.project.api.DispatcherServletWrapper"/>
                                        </property>
                                    </bean>
                                </list>
                            </property>
                            <property name="servletMappings">
                                <list>
                                    <bean class="org.eclipse.jetty.servlet.ServletMapping">
                                        <property name="pathSpecs">
                                            <list><value>/</value></list>
                                        </property>
                                        <property name="servletName" value="DefaultServlet"/>
                                    </bean>
                                </list>
                            </property>
                        </bean>
                    </property>
                </bean>
                <bean class="org.eclipse.jetty.server.handler.RequestLogHandler">
                    <property name="requestLog">
                        <bean class="org.eclipse.jetty.server.NCSARequestLog">
                            <constructor-arg value="/opt/company/logs/jetty-yyyy_mm_dd.log"/>
                            <property name="extended" value="false" />
                        </bean>
                    </property>
                </bean>
            </list>
        </property>
    </bean>
</property>
</bean>

I needed the dispatcher servlet to be aware of the ApplicationContext defining this Jetty server, so for reference my DispatcherServletWrapper class looks like this:

package com.company.project.api;

import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.web.context.WebApplicationContext;
import org.springframework.web.context.support.GenericWebApplicationContext;
import org.springframework.web.servlet.DispatcherServlet;

public class DispatcherServletWrapper extends DispatcherServlet implements ApplicationContextAware {

    private static final long serialVersionUID = -2281511575328213502L;
    private ApplicationContext appContext;

    @Override
    public void setApplicationContext(ApplicationContext arg0)
            throws BeansException {
        this.appContext = arg0; 
    }

    protected WebApplicationContext createWebApplicationContext(WebApplicationContext arg0) {
        GenericWebApplicationContext wac = new GenericWebApplicationContext();
        wac.setParent(appContext);
        wac.refresh();

        return wac;
    }

}
1
javaspring-mvcjettyembedded-jetty
I've made sure that my dev environment is using the exact same JRE as the production server, and that the IDE classpath matches exactly the classpath when run on the server. The server runs CentOS 5.9. I have discovered that the Jetty server works as expected, processes my requests and returns as expected, on a CentOS 6.4 server.... With the JRE and classpath being equal, my application being launched by nothing more fancy than "java -cp $(classpath variable) com.company.project.Main" what else is different that causes it to work in my Ubuntu dev environment and the 6.4 server, but not 5.9? - tdimmig
Not enough information to go by. You have not indicated how you start jetty. Can you include some code on how you start jetty? such as what you do with the Server instance? do you server.start() then server.join() or do you manage the server thread differently in your project? - Joakim Erdfelt
I switched to Jetty 8 (which required a different class for the Connector, and no other changes other than the library swap) and everything works. I still have no idea what perfect storm combination caused it to be broken with Jetty 9, which worked on several other platforms as I mentioned. But, backing up to Jetty 8 makes it work in my deployment platform so that's the direction I went. I'd close/delete this question if I knew how. Thanks for the comment though, I do appreciate someone showing some interest in helping. - tdimmig

1 Answers

1
votes

Ah, connectors have changed with Jetty 9.

Starting with Jetty 9, connectors are built-up with a chain of ConnectionFactory's, which provides connection type detection and fallbacks.

This was done to support the modern reality of alternative web connection types (such as TLS extensions, NPN, SPDY, WebSocket and even preparing for HTTP/2.0).

Here's some embedded examples:

Configuring a standard HTTP connector in Jetty 9

package org.eclipse.jetty.embedded;

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;

public class OneConnector
{
    public static void main(String[] args) throws Exception
    {
        // The Server
        Server server = new Server();

        // HTTP connector
        ServerConnector http = new ServerConnector(server);
        http.setHost("localhost");
        http.setPort(8080);
        http.setIdleTimeout(30000);

        // Set the connector
        server.addConnector(http);

        // Set a handler
        server.setHandler(new HelloHandler());

        // Start the server
        server.start();
        server.join();
    }
}

Configuring a connector chain for SSL + NPN + SPDY/3 + SPDY/2 + HTTPS in Jetty 9

package org.eclipse.jetty.embedded;

import org.eclipse.jetty.server.HttpConfiguration;
import org.eclipse.jetty.server.HttpConnectionFactory;
import org.eclipse.jetty.server.SecureRequestCustomizer;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.server.SslConnectionFactory;
import org.eclipse.jetty.spdy.server.NPNServerConnectionFactory;
import org.eclipse.jetty.spdy.server.SPDYServerConnectionFactory;
import org.eclipse.jetty.spdy.server.http.HTTPSPDYServerConnectionFactory;
import org.eclipse.jetty.spdy.server.http.ReferrerPushStrategy;
import org.eclipse.jetty.util.ssl.SslContextFactory;

public class SpdyConnector
{
    public static void main(String[] args) throws Exception
    {
        String jetty_home = System.getProperty("jetty.home","../../jetty-distribution/target/distribution");
        System.setProperty("jetty.home", jetty_home);

        // The Server
        Server server = new Server();

        // HTTP Configuration
        HttpConfiguration http_config = new HttpConfiguration();
        http_config.setSecureScheme("https");
        http_config.setSecurePort(8443);

        // HTTP connector
        ServerConnector http = new ServerConnector(server,new HttpConnectionFactory(http_config));        
        http.setPort(8080);
        server.addConnector(http);

        // SSL Context Factory for HTTPS and SPDY
        SslContextFactory sslContextFactory = new SslContextFactory();
        sslContextFactory.setKeyStorePath(jetty_home + "/etc/keystore");
        sslContextFactory.setKeyStorePassword("OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4");
        sslContextFactory.setKeyManagerPassword("OBF:1u2u1wml1z7s1z7a1wnl1u2g");

        // HTTPS Configuration
        HttpConfiguration https_config = new HttpConfiguration(http_config);
        https_config.addCustomizer(new SecureRequestCustomizer());

        // SPDY versions
        HTTPSPDYServerConnectionFactory spdy2 = 
            new HTTPSPDYServerConnectionFactory(2,https_config);

        HTTPSPDYServerConnectionFactory spdy3 = 
            new HTTPSPDYServerConnectionFactory(3,https_config,new ReferrerPushStrategy());

        // NPN Factory
        SPDYServerConnectionFactory.checkNPNAvailable();
        NPNServerConnectionFactory npn = 
            new NPNServerConnectionFactory(spdy3.getProtocol(),spdy2.getProtocol(),http.getDefaultProtocol());
        npn.setDefaultProtocol(http.getDefaultProtocol());

        // SSL Factory
        SslConnectionFactory ssl = new SslConnectionFactory(sslContextFactory,npn.getProtocol());

        // SPDY Connector
        ServerConnector spdyConnector = 
            new ServerConnector(server,ssl,npn,spdy3,spdy2,new HttpConnectionFactory(https_config));
        spdyConnector.setPort(8443);
        server.addConnector(spdyConnector);

        // Set a handler
        server.setHandler(new HelloHandler());

        // Start the server
        server.start();
        server.join();
    }
}