Reading gzipped HTTP response in Go

Question

I am trying to read a gzipped HTTP response with Go! but I always get the following error message :

panic: gzip: invalid header
[...] stack trace [...]

If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.

If I create a file with some dummy content and gzip it, I can read it from my Go! program.

The program I used for testing:

package main

import (
    "io"
    //"os"
    "fmt"
    "compress/gzip"
    "net/http"
)

func main() {
    /* This works fine
    f, _ := os.Open("/tmp/test.gz")
    defer f.Close()
    reader, err := gzip.NewReader(f)
    */

    // This does not :/
    resp, _ := http.Get("http://foo.com/")
    defer resp.Body.Close()
    reader, err := gzip.NewReader(resp.Body)

    if err != nil { panic(err) }

    buff := make([]byte, 1024)
    for {
        n, err := reader.Read(buff)

        if err != nil && err != io.EOF {
            panic(err)
        }

        if n == 0 {
            break
        }
    }

    s := fmt.Sprintf("%s", buff)
    fmt.Println(s)
}

Have I overlooked something ?

Stephen Weinberg Stephen Weinberg · Accepted Answer · 2012-10-30T00:58:22

EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.

client := new(http.Client)

request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")

response, err := client.Do(request)
defer response.Body.Close()

// Check that the server actually sent compressed data
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
    reader, err = gzip.NewReader(response.Body)
    defer reader.Close()
default:
    reader = response.Body
}

io.Copy(os.Stdout, reader) // print html to standard out

Error handling removed for brevity. I kept the defers.

Reading gzipped HTTP response in Go

3 Answers