http4s decompress gzipped http response

1
votes

I try decompress gzipped simple response on client side. What the appropriate way to do it with http4s?

import cats.effect.{Blocker, ContextShift, IO, Timer}
import java.util.concurrent._
import org.http4s.{Header, Headers, HttpVersion, Method, Request}
import org.http4s.client.{Client, JavaNetClientBuilder}
import org.http4s.implicits._

import scala.concurrent.ExecutionContext.global
implicit val cs: ContextShift[IO] = IO.contextShift(global)
implicit val timer: Timer[IO] = IO.timer(global)

val blockingPool = Executors.newFixedThreadPool(5)
val blocker = Blocker.liftExecutorService(blockingPool)
val httpClient: Client[IO] = JavaNetClientBuilder[IO](blocker).create
val uriYandex = uri"https://ya.ru"

val lstHeader: List[Header] =List(
  Header("Accept","text/plain")
  ,Header("Accept-Charset","utf-8")
  ,Header("Accept-Encoding","*")
)

val request2 = Request[IO](Method.GET, uriYandex, HttpVersion.`HTTP/2.0`, Headers(lstHeader))
val httpReq = httpClient.expect[String](request2)
val app = httpReq.map(resString => resString)

app.unsafeRunSync

http4s version "0.21.3" If I run it in IDEA Scala Worksheet. it works fine and output:

res0: String = <html class="i-ua_js_no i-ua_css_standart i-ua_browser_ i-ua_browser-engine_ i

But if I change ,Header("Accept-Encoding","*") into ,Header("Accept-Encoding","gzip") server return me gzipped content and output:

res0: String = " ??????????|?r?H??P?320???,Y??+d[v{??????" ?????%??9?'&6&66f???????3 ?{????7?/?

I have tried different decoders, but with this errors:

Failure(java.util.zip.ZipException: Not in GZIP format)

2
scalahttpgziphttp4s
FWIW, "Accept-Encoding: *" does not make any sense. What were you trying to achieve with that? - Julian Reschke
I can't see response headers in both cases, but looks like that with "Accept-Encoding: *" server not compress response and with gzip, response is compressed. - Aleksey N Yakushev
If you don't want compression, the right thing to send would be "Accept-Encoding: identity". See <greenbytes.de/tech/webdav/rfc7231.html#header.accept-encoding>. - Julian Reschke

2 Answers

2
votes

You can't just send the Accept-Encoding header, because when you do httpClient.expect[String], the HTTP client will try to decode the data as a string, and it doesn't know that it needs to decompress the data first.

Try using the GZip middleware.

1
votes

Thanks everybody. I solve my problem with next using of GZip.

import org.http4s.client.middleware.GZip
val gzClient = GZip()(httpClient)
val httpReq = gzClient.expect[String](request2)
val app = httpReq.map(resString => resString)