We are using WSO2 API Manager 1.10.0 and WSO2 Identity Server 5.2.0. We have created an inline script based API in API Manager and published it with Tier Availability as unlimited. We are able to invoke the API and consume the response. As part of experimentation, we intend to see how many concurrent requests can be catered.Using Jmeter, 1000 concurrent request were successfully inovked, but when testing for 2000 request, we encountered following error:-
> [2017-03-12 15:58:30,117] WARN - EndpointContext Endpoint :
> AnonymousEndpoint will be marked SUSPENDED as it failed [2017-03-12
> 15:58:30,118] WARN - EndpointContext Suspending endpoint :
> AnonymousEndpoint - last suspend duration was : 30000ms and current
> suspend duration is : 30000ms - Next retry after : Sun Mar 12 15:59:00
> UTC 2017 [2017-03-12 15:58:30,118] INFO - LogMediator STATUS =
> Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE =
> Unexpected error during sending message out [2017-03-12 15:59:12,858]
> ERROR - Axis2Sender Unexpected error during sending message out
> java.lang.IllegalStateException: I/O reactor has been shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:225)
> at org.apache.synapse.transport.passthru.connections.TargetConnections.getConnection(TargetConnections.java:98)
> at org.apache.synapse.transport.passthru.DeliveryAgent.submit(DeliveryAgent.java:156)
> at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:258)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:185)
> at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:167)
> at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
> at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:542)
> at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:79)
> at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:510)
> at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:372)
> at org.apache.synapse.endpoints.HTTPEndpoint.send(HTTPEndpoint.java:85)
> at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:111)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
> at org.apache.synapse.config.xml.AnonymousListMediator.mediate(AnonymousListMediator.java:30)
> at org.apache.synapse.mediators.filters.FilterMediator.mediate(FilterMediator.java:203)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
> at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
> at org.apache.synapse.rest.Resource.process(Resource.java:297)
> at org.apache.synapse.rest.API.process(API.java:378)
> at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97)
> at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:65)
> at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
> at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
> at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317)
> at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149)
> at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724) [2017-03-12 15:59:12,859] WARN - EndpointContext Endpoint : AnonymousEndpoint
> will be marked SUSPENDED as it failed [2017-03-12 15:59:12,859] WARN
> - EndpointContext Suspending endpoint : AnonymousEndpoint - last suspend duration was : 30000ms and current suspend duration is :
> 30000ms - Next retry after : Sun Mar 12 15:59:42 UTC 2017 [2017-03-12
> 15:59:12,859] INFO - LogMediator STATUS = Executing default 'fault'
> sequence, ERROR_CODE = 0, ERROR_MESSAGE = Unexpected error during
> sending message out [2017-03-12 15:59:34,318] INFO - LogMediator
> STATUS = Executing default 'fault' sequence, ERROR_CODE = 101504,
> ERROR_MESSAGE = Send timeout [2017-03-12 15:59:34,318] ERROR -
> PassThroughHttpSSLSender Failed to submit the response
> java.lang.NullPointerException
> at org.apache.synapse.transport.passthru.util.SourceResponseFactory.create(SourceResponseFactory.java:64)
> at org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:462)
> at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:267)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212)
> at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:493)
> at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:108)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
> at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
> at org.apache.synapse.mediators.MediatorFaultHandler.onFault(MediatorFaultHandler.java:86)
> at org.apache.synapse.FaultHandler.handleFault(FaultHandler.java:54)
> at org.apache.synapse.core.axis2.TimeoutHandler.processCallbacks(TimeoutHandler.java:188)
> at org.apache.synapse.core.axis2.TimeoutHandler.run(TimeoutHandler.java:88)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505) [2017-03-12 15:59:34,319] ERROR - Axis2Sender
> WSO2_AM_API_DEFAULT_VERSION:true,<?xml version='1.0'
> encoding='utf-8'?><soapenv:Envelope
> xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><am:fault
> xmlns:am="http://wso2.org/apimanager"><am:code>101504</am:code><am:type>Status report</am:type><am:message>Runtime
> Error</am:message><am:description>Send
> timeout</am:description></am:fault></soapenv:Body></soapenv:Envelope>
> Unexpected error sending message back org.apache.axis2.AxisFault:
> Failed to submit the response
> at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:610)
> at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:269)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212)
> at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:493)
> at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:108)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
> at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
> at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
> at org.apache.synapse.mediators.MediatorFaultHandler.onFault(MediatorFaultHandler.java:86)
> at org.apache.synapse.FaultHandler.handleFault(FaultHandler.java:54)
> at org.apache.synapse.core.axis2.TimeoutHandler.processCallbacks(TimeoutHandler.java:188)
> at org.apache.synapse.core.axis2.TimeoutHandler.run(TimeoutHandler.java:88)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
How can we increase the concurrent request count and what could be the reason behind this error? Do we have to modify API Manager configuration to accommodate around 5000 requests?
