Django: JSON Notifications using Redis PubSub, Node.js & Socket.io

19
votes

I came across this article: http://maxburstein.com/blog/realtime-django-using-nodejs-and-socketio/

Which has steered me in the somewhat right direction.

I currently have an iOS front end and a Django back end. I use Gunicorn to serve data to the front end application. The communication between my iOS application and my backed is REST based. I simply send JSON back and forth. I don't serve any webpages. Just JSON responses.

I've implemented a simple Post & Comment Model:

class Post(models.Model):
         user = models.ForeignKey(User)
         blog = models.CharField(max_length=5000)

class Comment(models.Model):
         comment = models.CharField(max_length=140)
         user = models.ForeignKey(User)
         post_id = models.ForeignKey(Post)
         created_at = models.DateTimeField(auto_now_add=True)

Users can make blog posts and other users can comment on them. So if userX has a blog post and userY comments on it. I would like to notify userX that userY commented on his/her post.

I used to rely on pyAPNS to notify users; a python wrapper that uses Twisted to send notifications to APNS, but if userX turns off push notifications for my app, then userX will not be able to receive in-app notifications. So I'm out of luck.

I only care about in-app notifications. I would still like userX to receive live updates while he is in the app.

Django can publish the message to a channel on Redis when a user makes a POST request. Node.js will subscribe to that channel and socket.io will send it out to that particular user.

Here is a stripped down version of my views.py where the comment object is created. I send the id the user who made the comment, the id of the post, and the id of the user who made the blog post. The user will make a post request to this url with json: http://example.com:8000/upload-comment/

def UploadComment(request):
         data  = json.loads(request.body)
         redis_server = redis.Redis(host='12.345.678.9', port=6379, db=0, password='mypassword')
         newComment = Comment()
         newComment.comment = data['comment']
         newComment.user_id = data['user_id']
         newComment.post_id = data['post_id']
         newComment.save() 
         PostOwner = data['post_owner_id'] #id of the blog post owner
         # Need to send a notification to PostOwner
         response_data = []
         response_data.append(
                 {'send_notifcation_to': PostOwner
                  'action': 'comment'
                  'comment_by': newComment.user.username)}
         redis_server.publish("notifications", json.dumps(response_data))
         return HttpResponse('Request Successful')

Node.js Implementation (According to Max Burstein's Article from Above)

var http = require('http');
var server = http.createServer().listen(4000);
var io = require('socket.io').listen(server);

And this is as far as I get :( I know this is quite sad but I'm left with many questions. How can I subscribe node.js to the remote Redis server that I published to from Django? How many clients can connect to this socket? is there limit? is a socket created for every client? or does every client listen in on the same socket? Can I send json data over this socket to one specific client? I know this is one big mammoth of post , but I'm in need of desperate help. If I wasn't clear with something, please do let me know so that I can edit the question. Thank you!

4
pythoniosdjangojsonnode.js
The article has a lot more of the node.js implementation than you've shown here. What about the rest of it isn't working for you? - Aaron Dufour
Are you locked to Node.js? I've implemented something analogous with python + celery + websockets. - Dwight Gunning
Do you want to stick with sockets and node.js? Or you would rather accept redis(pub/sub) + gunicorn + SSE (Server Sent Events)? - Aamir Adnan
@AamirAdnan SSE with Redis pub sub is another solution. But the problem, I do you know which client to specifically send it to? Only specific clients should receive notifications that pertain to them. Kind of like your Instagram activity feed. Every time someone comments or likes your shot, you'd be sent a notification in the browser or mobile device. - deadlock
Every user has its own channel, to which they are subscribed to. When user A performs action on User B activity you will send notification on User B channel. Whats difficult in identifying the channels? - Aamir Adnan

4 Answers

3
votes

I'd definitely not use node.js for this. You can handle websockets in Python just fine, with Celery or gevent or whatever.

Just create a thread which registers to Redis and listens for new messages.

When the user connects, put the socket into a weak-value hash indexed by the user's name; when a message for them arrives, look up the destination username in that hash and send it off. Weak-value because it cleans up after itself when the user disconnects. Simple, really.

I'd also send new messages to the server using the websocket instead of HTTP-PUT.

1
votes

Sounds like you should use something like RabbitMQ http://www.rabbitmq.com/devtools.html there's a node.js implementation and more.

Check it out, it should cheer you up :)

0
votes

I'm also using a wonderful library called django-websocket-redis.

Although it doesn't use gunicorn, it uses uWSGI, which shouldn't have too much overhead in migrating to. Just follow the docs, they're quite comprehensive.

https://django-websocket-redis.readthedocs.org/en/latest/

Taking some of your code and implementing it with django-websockets-redis:

## Make a json representation of your model available.

class Comment(models.Model):
     comment = models.CharField(max_length=140)
     user = models.ForeignKey(User)
     post_id = models.ForeignKey(Post)
     created_at = models.DateTimeField(auto_now_add=True)

    def as_json(self):
        return dict(
        id=self.id,
        comment=self.comment,
        created_at=self.created_at.strftime('%m/%d/%Y'),
        post_id=self.post_id
        user=self.user
        )

Then, in your views, when making the request, save a json representation to the redis facility you're subscribed to...you can do this anywhere really, either your serializer or view, but i'l leave that up to you.

def UploadComment(request):
     data  = json.loads(request.body)

     ## In django-websockets-redis, this portion below is registered in the settings.py file, therefore i am commenting this out. 
     ## redis_server = redis.Redis(host='12.345.678.9', port=6379, db=0, password='mypassword')

     newComment = Comment()
     newComment.comment = data['comment']
     newComment.user_id = data['user_id']
     newComment.post_id = data['post_id']
     newComment.save() 
     PostOwner = data['post_owner_id'] #id of the blog post owner
     # Need to send a notification to PostOwner

     ## Need to turn this into a string before posting to redis
     json_comment = str(newComment.as_json())

     ## Create a facility based on the PostOwner ID. The PostOwner will always be subscribed to this channel. 
     RedisPublisher(facility=PostOwner, broadcast=True).publish_message(json_comment)

     return HttpResponse('Request Successful')

Now that we've got the backend taken care of, on the front end visa vie JavaScript, your user subcribes to the Facility based on his user.id:

jQuery(document).ready(function($) {
    var ws4redis = WS4Redis({
    uri: '{{ WEBSOCKET_URI }}{{ PostOwner.id }}?subscribe-broadcast&publish-broadcast&echo',
    receive_message: receiveMessage,
    heartbeat_msg: {{ WS4REDIS_HEARTBEAT }}
});

// receive a message though the Websocket from the server
function receiveMessage(msg) {
    //This is where you would retrieve the JSON string from the Redis server, and subsequently manipulate the DOM with appends, Toastr Notifications, Etc...
}