GZIP with httpclient 4.5.3

Question

I am using httpclient 4.5.3 with the Fluent API (http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html). After upgrading to httpclient 4.5.3 from 4.2.x, the "Content-Encoding" header in the response appears to be dropped from the response header and I can't figure out how to support gzip compression.

I'm doing a GET request to https://www.yahoo.com and am sending the header "Accept-Encoding":"gzip"

My response header with 4.5.3 now shows the following with no Content-Encoding header:

Date: Thu, 01 Jun 2017 21:21:55 GMT
Strict-Transport-Security: max-age=2592000
X-Frame-Options: DENY
Set-Cookie: autorf=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=www.yahoo.com
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Age: 0
Transfer-Encoding: chunked
Connection: keep-alive
Via: http/1.1 ir11.fp.ne1.yahoo.com (ApacheTrafficServer)
Server: ATS
Cache-Control: no-store, no-cache, private, max-age=0
Expires: -1

My response handler has the following code. However entity.getContentEncoding() is always null.

 HttpEntity entity = response.getEntity();
            Header header = entity.getContentEncoding();
            if (header == null) {
                return EntityUtils.toString(entity);
            } else {
                return handleGzipStream(entity);
            }

Running through the debugger, it appears the wrappedEntity does have Content-Encoding:gzip in the header. How do I access that? Or what is the proper way of handling gzip compression with httpclient 4.5.3?

ok2c ok2c · Accepted Answer · 2017-06-02T08:57:02

ResponseContentEncoding removes content metadata headers such as Content-Length, Content-Encoding and Content-MD5 in order to ensure the content stream transparently decompressed by the client matches metadata embedded in the response message header. This is intentional. If you want to manually handle content decompression with fluent facade you need to create a custom request executor.

Executor executor = Executor.newInstance(HttpClients.custom().disableContentCompression().build());

GZIP with httpclient 4.5.3

1 Answers