I have a working cassandra cluster across two data centers. Each DC has 3 nodes with replication factor as 3 and READ/WRITE consistency as LOCAL_QUORUM.
I want to stop the traffic to a particular DC when two nodes in the DC are down, because quorum is no longer met. I expected this to be handled by my application(client) i.e. connect to other DC cassandra when local quorum is not met but it is not possible from there.
Can we setup some kind of rule at f5 load balancer to achieve this?
You can setup an external monitor on the BIG-IP to run a script determining cluster health and then load balance on the results. If you're using BIG-IP 11.x+ you create your script and import it, adding any needed arguments it may require. Then you create a monitor profile to call that external monitor.
If you have a DevCentral account, check out this page:
DevCentral Wiki: External Monitor
Scroll down and you'll see a ton of examples to build off. Examples to note are the MySql monitors. This is the path I would recommend for cluster health checks for BIG-IP.
Alternatively, you can simply query a web page looking for a success/failure message so if you already have a cluster health status page, you can have an HTTP monitor validate the message. You can customize the receive string to look for specific content or use regex to look for any specific string (such as clusterFailure or whatnot). From there, you can make the appropriate LB decisions. I ran a similar monitor that read a nagios status page and if it read a failure on a specific message, it would LB connections from that node.
