How to use nginx load balancing with rethinkdb proxy?

Question

This question was indeed started as question and while doing research I came up with this setup for nginx balancing for multiple rethinkdb proxies.

I have 3 rethinkdb proxy servers running. For sake of discussion lets call them:

proxy0
proxy1
proxy2

then all of then connect(--join) to first db in rethinkdb cluster. Now i i have defined nginx upstream like this:

upstream proxy {
    server proxy0:6003;      # rethindb proxy instance 0
    server proxy1:6003;      # rethinkdb proxy instance 1
    server proxy2:6003;      # rethinkdb proxy instance 2
}

and then just proxy pass that in location /.

This all looks great and simple but does not work. I am getting python error like this:

File "isDead.py", line 86, in <module>
    o = DeadLinksDelete(db="db", db_host="loadBalancerAddress", api_url=url, page=sys.argv[2])
  File "isDead.py", line 22, in __init__
    password="pass")
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/net.py", line 656, in connect
    return conn.reconnect(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/net.py", line 567, in reconnect
    return self._instance.connect(timeout)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/net.py", line 425, in connect
    self._socket = SocketWrapper(self, timeout)
  File "/usr/local/lib/python2.7/dist-packages/rethinkdb/net.py", line 325, in __init__
    raise ReqlDriverError(error)
rethinkdb.errors.ReqlDriverError: Connection is closed.

Ok, my first problem is that I am assuming that rethinkdb driver connects to proxy via http/https and as from my example that assumption is wrong.

Install nginx from sources to support --with-sream like this:

./configure --prefix=/opt/nginx --sbin-path=/usr/sbin/nginx --conf-path=/opt/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --with-threads --with-stream --with-http_slice_module --without-http_rewrite_module

Edit /opt/nginx/nginx.conf and add (after http{}):

stream {

server { listen 6003; proxy_pass db; }

upstream db { server proxy0:6003; server proxy1:6003; server proxy2:6003; }

}

Start nginx with just nginx
Set db host in you app to your nginx load balancer and that is that.

This setup is now working for me.

mlucy mlucy · Accepted Answer · 2016-07-16T20:25:52

Since the drivers don't communicate over HTTP, you'd probably be better off using something like HAProxy which is designed primarily to proxy TCP connections.

How to use nginx load balancing with rethinkdb proxy?

1 Answers