Timer overhead causing memory overflow

Question

From the timer.Stop() documentation

Stop prevents the Timer from firing. It returns true if the call stops the timer, false if the timer has already expired or been stopped. Stop does not close the channel, to prevent a read from the channel succeeding incorrectly.

I need to find a way to destroy timer object or channel created in program via After or NewTimer. I am not using these functions directly, just another library that implements timeout using these. The more requests I handle the program's memory keeps on increasing and then gets killed.

I have looked into the following places but without much help:

Please help, need to fix this desperately.

UPDATE

The suspected code is at https://github.com/gocql/gocql/blob/986e33a705412161497203d55d0669d04282f5ff/conn.go#L546

var timeoutCh <-chan time.Time
if c.timeout > 0 {
    timeoutCh = time.After(c.timeout)
}

select {
case err := <-call.resp:
    if err != nil {
        if !c.Closed() {
            // if the connection is closed then we cant release the stream,
            // this is because the request is still outstanding and we have
            // been handed another error from another stream which caused the
            // connection to close.
            c.releaseStream(stream)
        }
        return nil, err
    }
case <-timeoutCh:
    close(call.timeout)
    c.handleTimeout()
    return nil, ErrTimeoutNoResponse
case <-c.quit:
    return nil, ErrConnectionClosed
}

How do I know this ? I ran go tool pprof to capture memprof and what it showed is :

So you're saying the references to these hundred of thousands of timer channels are not going out of scope, so they don't get garbage collected? — Filip Haglund
Hi all I have added the suspected code and pprof memprofile output to question. — user568109
That code isn't incorrect, you don't need to explicitly clean up a timer (time.After is just shorthand for time.NewTimer(d).C). How long is your timeout set for, and what is the request rate? — JimB
@JimB Timeout is set to 15 min and request rate is > 3K per second (as fast as it can take). I have tried replacing time.After with time.NewTimer and using Stop(). It is leaner but still runs out of memory after some time. — user568109

FEiN FEiN · Accepted Answer · 2016-02-20T15:47:26

I've merged a fix for this, https://github.com/gocql/gocql/pull/661 please raise an issue if you run into further issues

Timer overhead causing memory overflow

1 Answers