Go HTTP Request POST headers, wait for response, then post chunked body content

Question

I'm trying to figure out if there is a way to post chunked data to a HTTP server which attempts to send headers before accepting any of my request body.

I have a server which receives never ending streams of data via POST requests. Upon receiving new POST requests, it constructs headers and immediately attempts to Flush.

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    requestId := uuid.Must(uuid.NewV4()).String()

    w.Header().Set("X-Request-Id", requestId)
    w.Header().Set("Content-Type", "application/octet-stream")

    client := &Connection{requestId, make(chan []byte, 50), r, w}
    defer client.Request.Body.Close()

    switch client.Request.Method {
        case http.MethodPost:
            client.Writer.(http.Flusher).Flush()
            // goes on to read data from the connection until it is closed
    }
}

This works fine for POST clients which are implemented using sockets, which can decide to read the servers response before actually sending any of the body data. However, I can't figure out how to get it to work for a Go http Request object.

On the client side, I have a goroutine which is Writing data to an io.PipeWriter, and I have passed the PipeReader (pr) to the Request object:

req := &http.Request{
    Method: "POST",   
    ProtoMajor: 1,    
    ProtoMinor: 1,    
    URL: serverUrl,
    TransferEncoding: []string{"chunked"},
    Body: pr,
    Header: make(map[string][]string),
}

resp, err := http.DefaultClient.Do(req)

This hangs forever on both ends because the PipeReader is never closed, and the server blocks on Flush. What I would like to do is send headers, wait for the response from the server, then start sending body content.

I have tried simply not including the Body in the Request object, this sends the headers, receives the response headers, AND the connection seems to stay open. But from all my reading I can't seem to find a way to send additional data over this connection.

Of course, I can just not Flush on the server end and handle the data - but I'm actually implementing a protocol which specifies this behavior, and so existing client software doesn't send the body data unless it first receives the response headers.

After sending a request, yes, the connection will (often) stay open, for subsequent requests. But that won't help you here. For the level of control you seem to need, I don't think the net/http package in the std lib is sufficient. — Flimzy
I have the feeling that you are trying to use the syntax of HTTP but with different semantics. The semantics of HTTP don't have the idea of sending part of the request, reading part of the response, sending more data from the request etc. One way to do this is to have multiple request/responses. Another way might be to use a preliminary response (status code 100) after receiving the request header to signal the client to go on with the request body and then send a final response once the body is done (don't know if this is supported in golang). Or you could use Websockets. — Steffen Ullrich
That's correct, Steffen. As noted in the post, I'm implementing a protocol which is basically just a behavioral specification for HTTP streaming. If there's a simple way to do this without the http library I'm all ears. It would be really handy if I could just use this library for constructing and sending the request with an empty body, and then somehow get access to the still open connection object. But it doesn't seem like that's possible, and I'm not looking forward to implementing http over sockets manually to solve 5% of the problem. — unclemeat

unclemeat unclemeat · Accepted Answer · 2019-02-06T23:46:43

So the server is blocking in Flush because the body is still open. Turns out if you set the Connection header to close in the client's POST request (EDIT: You can set it in the servers response headers before flushing and it works regardless of what the client is doing), the server will ignore the fact that the body is still open and flush anyway (and keep the connection open to read data). Neat.

I can't seem to find any documentation in either the HTTP spec or the Go http library which describes this behavior. But here we are.

Go HTTP Request POST headers, wait for response, then post chunked body content

1 Answers