Redis pubsub cause Too Many Files Open Error

Question

I have a server used Redis and Socket.IO. If a user connect to the socket.IO, server will run a greenlet and subscribe to redis channel. If the user disconnect, the greenlet will unsubscribe the channel.

After a lot of connection and disconnection, I use 'client list' command in redis-cli, I found that the number of unsubscribe is additive and it will never closed like other command. And it will finally cause 'Too Many Files Open Error'

My Code is

class ListenMsgThreading(Greenlet):
    def __init__(self, app_eui, request_sid):
        Greenlet.__init__(self)
        self.ps = redis_db.pubsub()
        self.ps.subscribe('channel')

    def stop(self):
        self.ps.unsubscribe('channel')

    def run(self):
        for item in self.ps.listen():
            if item is not None:
                print(item)

I notice that the reason may be

timeout only applies to number clients and it does not apply to Pub/Sub clients, since a Pub/Sub connection is a push style connection so a client that is idle is the norm.

Should I implement the timeout for unsubscribe command? Or use other method to stop subscribe?

Paul Becotte Paul Becotte · Accepted Answer · 2016-06-29T02:57:53

Looking at the code

https://github.com/andymccurdy/redis-py/blob/master/redis/client.py#L2281

I can guess at the issue. It looks like the PubSub object checks a connection out of the connection pool on the subscribe command, but doesn't release it on unsubscribe. It SHOULD release it when your greenlet goes out of scope (the del method)- are you holding them in scope for some reason so that they don't get freed? This will lead to memory issues either way...

however, you could try using the reset method to force release that connection without messing with scope issues.

self.ps.unsubscribe()
self.ps.reset()

Redis pubsub cause Too Many Files Open Error

1 Answers