WSO2 ESB proxy service does not start if target is unavailable

Question

I have a simple Proxy Service deployed on WSO2 ESB. The service is defined as follow:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="HelloWorld"
       transports="https http"
       startOnLoad="false"
       trace="enable"
       statistics="enable">
   <description/>
   <target>
      <endpoint>
         <wsdl service="HelloWorld"
               port="HelloWorldImplPort"
               uri="http://myhostname:9001/HelloWorldInternal?wsdl">
            <suspendOnFailure>
               <errorCodes>101508,101505,101503,101504</errorCodes>
               <initialDuration>1000</initialDuration>
               <progressionFactor>1.1</progressionFactor>
               <maximumDuration>30000</maximumDuration>
            </suspendOnFailure>
         </wsdl>
      </endpoint>
      <outSequence>
         <log level="full"/>
         <send/>
      </outSequence>
   </target>
   <publishWSDL uri="http://myhostname:9001/HelloWorldInternal?wsdl"/>
</proxy>

When I start WSO2 ESB when the target service is down, I get the following error in ESB logs:

[2014-02-20 14:25:05,168]  INFO - ProxyService Building Axis service for Proxy service : HelloWorld
[2014-02-20 14:25:06,176] ERROR - ProxyService Error reading from wsdl URI
java.net.ConnectException: Connection refused: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:75)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
    at sun.net.www.http.HttpClient.New(HttpClient.java:300)
    at sun.net.www.http.HttpClient.New(HttpClient.java:316)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:928)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1296)
    at org.apache.synapse.config.SynapseConfigUtils.getOMElementFromURL(SynapseConfigUtils.java:312)
    at org.apache.synapse.core.axis2.ProxyService.buildAxisService(ProxyService.java:284)
    at org.apache.synapse.Axis2SynapseController.deployProxyServices(Axis2SynapseController.java:709)
    at org.apache.synapse.Axis2SynapseController.createSynapseEnvironment(Axis2SynapseController.java:388)
    at org.apache.synapse.ServerManager.start(ServerManager.java:181)
    at org.wso2.carbon.mediation.initializer.ServiceBusInitializer.initESB(ServiceBusInitializer.java:424)
    at org.wso2.carbon.mediation.initializer.ServiceBusInitializer.activate(ServiceBusInitializer.java:182)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.event.core.internal.builder.EventBrokerHandler.startEventBroker(EventBrokerHandler.java:58)
    at org.wso2.carbon.event.core.internal.builder.EventBrokerBuilderDS.activate(EventBrokerBuilderDS.java:72)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
    at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:517)
    at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
    at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:347)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
    at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1267)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1186)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1081)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5027)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
[2014-02-20 14:25:06,217]  WARN - Axis2SynapseController The proxy service HelloWorld cannot be deployed - Continue in Proxy Service fail-safe mode.
[2014-02-20 14:25:06,217]  INFO - Axis2SynapseController Deploying EventSources...
[2014-02-20 14:25:06,227]  INFO - ServerManager Server ready for processing...

The service now disappears from WSO2 web console.

1. How can I make the service not disappear from the console?
2. How can I configure WSO2 to automatically start the proxy service when my target service is up?

Now, any call to the proxy service result in HTTP 202 Accepted, even when the target service is finally up. The only way to make my proxy service visible again, is to touch it's configuration file, so it gets redeployed.

I'm using WSO2 ESB 4.8.1

Answer 1

I have also encountered similar issue. ESB tries to read the WSDL at the startup.

Please create a JIRA here: https://wso2.org/jira/browse/ESBJAVA

Touching the proxy config will re-deploy it. I also do not know any other workaround to re-deploy.

One workaround is to save your WSDL in the registry and refer that WSDL in your proxy configuration. This answer might help for you.

Answer 2

So I have created Local entry named HelloWorld.wsdl of type Inline XML. Then, I'm referencing it in my HelloWorld.xml service definition like this: <publishWSDL key="HelloWorld.wsdl"/>, but still

1. ESB complains at startup about unavailable target service
2. Each attempt to get WSLD from ESB is passed to the target WebService (I can see it in my WS logs)

Here is my full service definition

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="HelloWorld"
       transports="https http"
       startOnLoad="false"
       trace="enable"
       statistics="enable">
   <description/>
   <target>
      <endpoint>
         <wsdl service="HelloWorld"
               port="HelloWorldImplPort"
               uri="http://myhostname:9001/HelloWorldInternal?wsdl">
            <suspendOnFailure>
               <errorCodes>101508,101505,101503,101504</errorCodes>
               <initialDuration>1000</initialDuration>
               <progressionFactor>1.1</progressionFactor>
               <maximumDuration>30000</maximumDuration>
            </suspendOnFailure>
         </wsdl>
      </endpoint>
      <outSequence>
         <log level="full"/>
         <send/>
      </outSequence>
   </target>
   <publishWSDL key="HelloWorld.wsdl"/>
</proxy>

On the contrary, having another proxy service, for unavailable target service, but without WSDL, gives an error [2014-02-21 09:59:19,250] WARN - WSDLEndpointFactory Could not connect to the WSDL endpoint http://someipaddress:28280/services/HelloWorldExternal?wsdl java.net.ConnectException: Connection refused: connect

but the proxy service gets deployed. The proxy service configuration follows:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="HelloWorldExternal2"
       transports="https http"
       startOnLoad="true"
       trace="enable"
       statistics="enable">
   <target>
      <endpoint>
         <wsdl service="HelloWorldExternal"
               port="HelloWorldExternalHttpSoap11Endpoint"
               uri="http://someipaddress:28280/services/HelloWorldExternal?wsdl"/>
      </endpoint>
      <outSequence>
         <log level="full"/>
         <send/>
      </outSequence>
   </target>
   <!-- publishWSDL uri="http://someipaddress:28280/services/HelloWorldExternal?wsdl"/ -->
</proxy>

How can I publish WSDL, and make the proxy service load at startup, even though the target service is unavailable?

EDIT 2014-02-21 11:01

For the ESB to pick custom WSDL from local entries, one have to set proxy service parameter useOriginalwsdl to true. This solves service not starting at startup. But, doing so, rises another issue. The proxy service still can't be called. Server logs are attached below, and client fails with socket read timeout. I'm aware of suspendOnFailure settings, which should expire connectivity errors within 30 seconds.

[2014-02-21 10:52:06,121] ERROR - ClientUtils The system cannot infer the transport information from the /services/HelloWorld.HelloWorldHttpSoap11Endpoint URL.
[2014-02-21 10:52:06,121] ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault: The system cannot infer the transport information from the /services/HelloWorld.HelloWorldHttpSoap11Endpoint URL.
        at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)
        at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:115)
        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
        at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:482)
        at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:59)
        at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:338)
        at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:333)
        at org.apache.synapse.endpoints.WSDLEndpoint.send(WSDLEndpoint.java:69)
        at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:187)
        at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
        at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:411)
        at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
        at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
[2014-02-21 10:52:06,126]  WARN - FaultHandler ERROR_CODE : 0
[2014-02-21 10:52:06,126]  WARN - FaultHandler ERROR_MESSAGE : Unexpected error during sending message out
[2014-02-21 10:52:06,126]  WARN - FaultHandler ERROR_DETAIL : org.apache.synapse.SynapseException: Unexpected error during sending message out
        at org.apache.synapse.core.axis2.Axis2Sender.handleException(Axis2Sender.java:172)
        at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:71)
        at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:338)
        at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:333)
        at org.apache.synapse.endpoints.WSDLEndpoint.send(WSDLEndpoint.java:69)
        at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:187)
        at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
        at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:411)
        at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
        at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.axis2.AxisFault: The system cannot infer the transport information from the /services/HelloWorld.HelloWorldHttpSoap11Endpoint URL.
        at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)
        at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:115)
        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
        at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:482)
        at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:59)
        ... 11 more

[2014-02-21 10:52:06,129]  WARN - FaultHandler ERROR_EXCEPTION : org.apache.synapse.SynapseException: Unexpected error during sending message out
[2014-02-21 10:52:06,129]  WARN - FaultHandler FaultHandler : AnonymousEndpoint

Right now, it seems to be no option to start WSO2 ESB proxy service when target service is down.