Spring WebClient with proxy server: Missing HTTP headers

Question

I'm using the Spring WebClient from spring-boot-starter-webflux 2.1.3.RELEASE to check the anonymity level of proxy servers. After I made some requests with the WebClient to a custom node.js http server through some proxy servers, there are no proxy related HTTP headers in my requests. I'm missing e.g. x-forwarded-for, via, x-proxy-id.. just the remote-address is exposed.

As far as I understood it, the netty client connects to the proxy via tcp CONNECT for every proxy type (HTTP,SOCKS4/5) and this is the reason for the missing headers.

Question: Is there a way to get classic proxy HTTP headers with this approach or is there another way to configure a proxy server with the WebClient?

My example configuration:

HttpClient httpClient = HttpClient.create()
    .tcpConfiguration(tcpClient ->
        tcpClient
            .proxy(proxy -> {                       
                proxy.type(ProxyProvider.Proxy.HTTP).address(new InetSocketAddress(ip, port));})
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
            .doOnConnected(connection ->
                connection
                    .addHandlerLast(new ReadTimeoutHandler(10000))
                    .addHandlerLast(new WriteTimeoutHandler(10000))));

ReactorClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);

WebClient build = WebClient.builder()
    .baseUrl(baseUrl)
    .clientConnector(connector)
    .build();

Expected headers

{"user-agent":"ReactorNetty/0.8.4.RELEASE","host":"21X.8X.XX.145:8XX","accept":"*/*","x-proxy-id":"719306848","x-forwarded-for":"21X.8X.XX.145","via":"1.1 101.XX.8X.11X (Mikrotik HttpProxy)"}

Actual headers

{"user-agent":"ReactorNetty/0.8.4.RELEASE","host":"21X.8X.XX.145:8XX","accept":"*/*"}

I don't understand your question. Those headers are added by the proxy itself. How are you checking that those headers exist on the server side? Could you show the code snippet you've used? — Brian Clozel
Yes, the headers should be added by the proxy, but they are not present when I make a request with the webclient. I've tested the requests against a small node.js server, that returns the headers. Headers with webclient: {"user-agent":"ReactorNetty/0.8.4.RELEASE","host":"21X.8X.XX.145:8XX","accept":"*/*"} Headers with another client, e.g. insomnia: {"host":"21X.8X.XX.145:8XX","user-agent":"insomnia/6.3.2","accept":"*/*","x-proxy-id":"719306848","x-forwarded-for":"21X.8X.XX.145","via":"1.1 101.XX.8X.11X (Mikrotik HttpProxy)"} — m.w.
@DenissM. unfortunately not, I switched to a node.js client. Was my question maybe too incomprehensible? — m.w.
@BrianClozel maybe you could help us out? Is the question still unclear for you? — m.w.

m.w. m.w. · Accepted Answer · 2019-07-17T07:55:07

The reason for the missing headers is the CONNECT request, which establishes a TCP tunnel instead of sending HTTP requests to the proxy.

But unfortunately as stated in

https://github.com/netty/netty/issues/8269

and

https://github.com/spring-projects/spring-framework/issues/21767

the underlying netty client which is used by the webclient issuing a CONNECT request for every type of proxy and this will not be changed.

Spring WebClient with proxy server: Missing HTTP headers

1 Answers