Heroku H27 Client Request Interrupted on Server Sent Events (SSE) GET event

Question

I have a node.js server that uses SSE to send updates to attached clients. Occasionally, I get a server error H27. At the same time, other client requests get lost probably while the client is re-registering to the SSE event service.

The time elapsed between the client GET /event request and the server H27 error is anywhere between 13sec to 19:35min (in 30 different occurrences encountered). But there is a full correlation between the timing of the GET /event request and a corresponding H27 error. I am sending a keep alive message from the server every 50sec to workaround the Heroku timeout limit of 55sec.

Here is an example for the full warning I get in Heroku logs: 2020-10-17T08:49:04.525770+00:00 heroku[router]: sock=client at=warning code=H27 desc="Client Request Interrupted" method=GET path="/event" host=appname.herokuapp.com request_id=c4c4e2fd-16ca-4292-a27b-2a70b12b16fa fwd="77.138.167.76" dyno=web.1 connect=1ms service=9499ms status=499 bytes= protocol=https

that resulted from the following GET request: 2020-10-17T08:48:55.027638+00:00 app[web.1]: Client 8 registered

Any idea how I can overcome this? My problem is that my application heavily relies on SSE, and if I now must switch to another mechanism (e.g., socket), it will take a considerable effort.

EDIT On further investigation, this seems to occur due to the client being unable to STAY connected to the Server Sent Events route on Heroku servers. While it can establish the first connection, it cannot stay connected. I suspect it has to do something with Heroku Request Timeouts and the way Heroku handles routing in general.

I still have not found a solution for this problem so everyone please feel free to comment.

Answer 1

I have found a very hacky solution to this problem that works for me since I am the only user of the application.

First, some context. The problem is that for some reason apps deployed on Heroku, while allowing clients to connect to the SSE route successfully, are unable to detect the state of the client connection. After the client connects, within a short time period of the server being unable to detect the client, a code=H27 desc="Client Request Interrupted" error is posted on the server, which it takes as a sign that the client connection has closed. Since it thinks the client connection has closed, it naturally fires a close event resulting in the execution of code in the req.close block within the SSE route.

On my dev environment, my SSE route is configured as follows:

router.get('/updates', (req, res) => {
    res.writeHead(200, {
        'Content-Type': 'text/event-stream',
        'Cache-Control': 'no-cache',
        'Connection': 'keep-alive'
    })

    res.write('\n')

    // eventEmitted listeners are added for each connected client.
    const eventEmitted = (data) => {
        res.write(`data: ${JSON.stringify(data)}\n\n`)
    }

    eventEmitter.addListener('event', eventEmitted)

    // eventEmitted listeners are removed when each client disconnects.
    req.on('close', () => {
        eventEmitter.removeAllListeners()
    })
})

However, as mentioned earlier Heroku fires the close event soon after the client has connected due to its inability to detect the state of the client connections, resulting in eventEmitter.removeAllListeners() being executed. Naturally, my client receives no notifications thereafter. I have changed my route as follows so it works on Heroku:

// Server Sent Events route for receiving realtime notifications when events are emitted.
router.get('/updates', (req, res) => {
    // eventEmitted listeners are added for the most recently connected client.
    const eventEmitted = (data) => {
        res.write(`data: ${JSON.stringify(data)}\n\n`)
    }

    res.writeHead(200, {
        'Content-Type': 'text/event-stream',
        'Cache-Control': 'no-cache',
        'Connection': 'keep-alive'
    })

    res.write('\n')

    eventEmitter.removeAllListeners()
    eventEmitter.addListener('event', eventEmitted)
})

The result is that since req.on('close') is removed, no H27 error occurs on Heroku since it does not try to detect client state. The disadvantage is that only the most recently connected client can receive realtime updates from the server. All previously connected clients will not have an event listener attached to deliver updates to them. This works for me since I am the only user of the application but will not work if you have more than 1 client or more than 1 instance of the app running.

The other option is to remove the req.on('close') altogether and never remove any added listeners. This allows all your clients to receive updates at the cost that your server will have a lot of orphan event listeners for every disconnected client, eventually resulting in a memory leak.

This is not really an acceptable solution for most people but the only one I could come up with. If anyone has a better solution, please feel free to post it here.