Chrome DevTools Protocol - ContinueInterceptedRequest with gzip body in Golang

Question

I have been working on a golang script that uses the chrome devtools protocol to:

1) Intercept a request

2) Grab the response body for the intercepted request

3) Make some modifications to the html document

4) Continue the intercepted request

The script works for HTML documents except when Content-Encoding is set to gzip. The step-by-step process looks like this"

1) Intercept Request

 s.Debugger.CallbackEvent("Network.requestIntercepted", func(params godet.Params) {
    iid := params.String("interceptionId")
    rtype := params.String("resourceType")
    reason := responses[rtype]
    headers := getHeadersString(params["responseHeaders"])

    log.Println("[+] Request intercepted for", iid, rtype, params.Map("request")["url"])
    if reason != "" {
        log.Println("  abort with reason", reason)
    }

    // Alter HTML in request response
    if s.Options.AlterDocument && rtype == "Document" && iid != "" {
        res, err := s.Debugger.GetResponseBodyForInterception(iid)

        if err != nil {
            log.Println("[-] Unable to get intercepted response body!")
        }

        rawAlteredResponse, err := AlterDocument(res, headers)
        if err != nil{
            log.Println("[-] Unable to alter HTML")
        }

        if rawAlteredResponse != "" {
            log.Println("[+] Sending modified body")

            err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
            if err != nil {
                fmt.Println("OH NOES AN ERROR!")
                log.Println(err)
            }
        }
    } else {
        s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), "", "", "", "", nil)
    }
})

2) Alter the response body

Here I am making small changes to the HTML markup in procesHtml() (but the code for that function is not relevant to this issue, so will not post it here). I also grab headers from the request and when necessary update the content-length and date before continue the reponse. Then, I gzip compress the body when calling r := gZipCompress([]byte(alteredBody), which returns a string. The string is then concatenated to the headers so I can craft the rawResponse.

func AlterDocument(debuggerResponse []byte, headers map[string]string) (string, error) {
    alteredBody, err := processHtml(debuggerResponse)
    if err != nil {
        return "", err
    }


    alteredHeader := ""
    for k, v := range headers{
        switch strings.ToLower(k) {
            case "content-length":
                v = strconv.Itoa(len(alteredBody))
                fmt.Println("Updating content-length to: " + strconv.Itoa(len(alteredBody)))
                break
            case "date":
                v = fmt.Sprintf("%s", time.Now().Format(time.RFC3339))
                break
        }
        alteredHeader += k + ": " + v + "\r\n"
    }

    r := gZipCompress([]byte(alteredBody))

    rawAlteredResponse := 
    base64.StdEncoding.EncodeToString([]byte("HTTP/1.1 200 OK" + "\r\n" + alteredHeader + "\r\n\r\n\r\n" + r))

    return rawAlteredResponse, nil
}

Note: I am now gzip compressing the body for all responses. The above is temporary while I figure out how to solve this issue.

The gzip compress function looks like this:

func gZipCompress(dataToWorkWith []byte) string{
    var b bytes.Buffer

    gz, err := gzip.NewWriterLevel(&b, 5)
    if err != nil{
        panic(err)
    }
    if _, err := gz.Write(dataToWorkWith); err != nil {
        panic(err)
    }
    if err := gz.Flush(); err != nil {
        panic(err)
    }
    if err := gz.Close(); err != nil {
        panic(err)
    }
    return b.String()
}

As seen in the first code snippet, the response body and headers are set here:

err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)

The result is a bunch of garbled characters in the browser. This works without the gzip functions for non gzipped requests. I have changed the compression level as well (without success). Am I processing the body in the wrong order (string > []byte > gzip > string > base64)? Should this be done in a different order to work? Any help would be immensely appreciated.

The response looks like this, which Chrome puts inside a <body></body> tag

����rܸ� ��_A��Q%GH��Kʔ��vU�˷c�v�}

or in the response:

I can also tell that it is compressing correctly as, when I remove headers, the request results in a .gz file download with all the correct .html when uncompressed. Additionally, the first few bytes in the object returned in gZipCompress tell me that it is gzipped correctly:

31 139 8

or

0x1f 0x8B 0x08

Answer 1

I ended up using a different library that handles larger responses better and more efficiently.

Now, it appears that the DevTools protocol returns the response body after decompression but before rendering it in the browser when calling Network.GetResponseBodyForInterception. This is an assumption only of course, as I do not see code for that method in https://github.com/ChromeDevTools/devtools-protocol. The assumption is based on the fact that, when calling Network.GetResponseBodyForInterception the response body obtained is NOT compressed (though it may be base64 encoded). Furthermore, the method is marked as experimental and the documentation does not mention anything in regards to compressed responses. Based on that assumption, I will further assume that, at the point that we get the response from Network.GetResponseBodyForInterception it is too late to compress the body ourselves. I confirm that the libraries that I am working with do not bother to compress or uncompress gzipped responses.

I am able to continue working with my code without a need to worry about gzip compressed responses, as I can alter the body without problems.

For reference, I am now using https://github.com/wirepair/gcd, as it is more robust and stable when intercepting larger responses.