Camel-Jetty http proxy + large response data casue issue: Buffering capacity exceeded

3
votes

jetty expert, I got error when getting back large data via camel-jetty as http proxy, any experience/idea/suggestion are welcome, thanks in advance. RouteBuilder code: 

from("servlet:balancer?matchOnUriPrefix=true")
.to("jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?" + "bridgeEndpoint=true"
                                            + "&disableStreamCache=true"
                                            + "&throwExceptionOnFailure=false" 
                                            + "&bufferSize=1024000000"
                                            + "&inboundBufferSize=1024000000"
                                            + "&outboundBufferSize=1024000000"
                                            + "&eagerCheckContentAvailable=true"
                                            + "&httpClient.responseBufferSize=1024000000&responseBufferSize=1024000000&chunked=false")

ENV:

<jdk.version>1.8</jdk.version>
<camel.version>2.17.0</camel.version>
<spring.version>4.2.4.RELEASE</spring.version>
<spring.security.version>4.0.4.RELEASE</spring.security.version>

Got error when http based rest-service return back 5000 records of json data, I have tried many ways to adjust the response buffer size, unfortunately still not working, it works when service side return <1000 records: 

    [2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@61e4dcc8[p=40262,l=47776,c=1024000000,r=7514]={umn12":"data4782"...4875","\r\n2000\r\n<<<column10":"data48...16":"data4892",>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=47776,l=47776,c=1024000000,r=0]={umn12":"data4782"...16":"data4892",<<<>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         142] - filled 31280 SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,3/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        113] - Read 31280 bytes DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} from SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,0/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@5991d3e4[p=0,l=678,c=1024000000,r=678]={<<<"column17":"data4...a4894","column8>>>\r\n2000\r\n":"data48...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CHUNKED_CONTENT 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNKED_CONTENT --> CHUNK_SIZE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK_SIZE --> CHUNK 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@2b0b1218[p=686,l=8878,c=1024000000,r=8192]={"column17":"data4...column8\r\n2000\r\n<<<":"data4894","col..."column6":"data>>>\r\n2000\r\n4913","co...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        228] - Failed HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: req=false/rsp=true java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1555] - close HttpParser{s=CHUNK,2105344 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CLOSED 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        514] - Response failure HttpResponse[HTTP/1.1 200 OK]@54e3183e HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 on HttpChannelOverHTTP@4b2502b4(exchange=HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        529] - Concurrent failure: response termination skipped, performed by helpers 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        259] - Aborted (true) while active HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        541] - State update failed: TRANSIENT -> CONTENT: FAILURE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        210] - Terminated response for HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497, result: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        442] - Response complete HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpChannel                         89] - HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 disassociated true from HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.ConnectionPool                      257] - Connection removed HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] - pooled: 0 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         75] - oshut SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         154] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] oshut 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         115] - close SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.SelectorManager                         756] - Destroyed SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractConnection                      210] - onClose HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         157] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] closed 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        450] - Request/Response failed: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.DefaultErrorHandler               153] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). On delivery attempt: 0 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.Pipeline                          78] - Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8] Exception: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [ERROR] [org.apache.camel.processor.DefaultErrorHandler               204] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[route1            ] [route1            ] [servlet:/balancer?matchOnUriPrefix=true                                       ] [      3135]
[route1            ] [policy1           ] [policy[null]                                                                  ] [      3136]
[route1            ] [to1               ] [jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?bridgeEndpoint=true&d] [      3133]

Stacktrace
--------------------------------------------------------------------------------------------------------------------------------------- 
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
    at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155)
    at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:360)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:254)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:38)
    at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1499)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1272)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
    at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
    at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:245)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
    at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
    at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:274)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:322)
    ... 13 more
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed true, remaining 22402 HttpParser{s=CLOSED,2105344 of -1}
1
apache-camelhttp-proxyjetty-9
I don't know how to get Jetty to not buffer anything. Not sure it is possible. Can you try the http4 component as the producer in your proxy route? In Camel 2.17.0 it comes with the added benefit that it won't cache the response stream if stream caching is disabled. - Ralf
Due to some other reason, I need to use camel-jetty, anyway thank you, I'm going to figure out how to cache nothing, that's the first choice :), modify response cache buffer size as secondary... - Lori Sun

1 Answers

3
votes

The reason is: camel-jetty9-2.17.0.jar has a limitation, org.apache.camel.component.jetty9.JettyContentExchange9#send

BufferingResponseListener responseListener = new BufferingResponseListener() {

    @Override
    public void onComplete(Result result) {
        if (result.isFailed()) {
            doTaskCompleted(result.getFailure());
        } else {
            onResponseComplete(result, getContent(), getMediaType());
        }
    }
};

In BufferingResponseListener there is default maxLength is 2MB(2 * 1024 * 1024). And single parameter constructor: BufferingResponseListener(int maxLength) is only way to adjust maxLength. but camel use a default constructor, so all response larger than 2MB will get such error.