mariadb connector J Aurora Fast failover implemantation

0
votes

I'm trying to understand Aurora fast failover implementation of Mariadb Connetor J. Although I am not java apps engineer, my prime work is jobs as DBA. I think I have little knowledge about OOP languages and have read the source code of mariadb connector J focusing related Aurora implementation. But it was difficult and cannot have confidence about my guess.

I really appreciate if share your knowledge about it or just any few comments.

In the latest version, we just register cluster endpoint of Aurora cluster and driver automatically find out every instance endpoint.

My guess how this logic work is

  • Driver generates every endpoint connect string from information_schema.REPLICA_HOST_STATUS that knows all instance identifiers on Sever_id column. There is a pattern about endpoint string. So once driver succeeded to connect any instance from cluster endpoint, driver can generate every instance endpoint.

  • After git every instance endpoints, driver throws a query "SHOW GLOBAL STATUS LIKE 'innodb_read_only'. If return value is 0(false), it set as Writer, otherwise set as Reader.

  • Driver push the connect string into "blacklist" if health check fails.( I cannnot find where health check is written though)

  • Driver tries to connect from not black-listed connect string, but if it fails, it tries black-listed connect string.

My guess is mainly from below java files.

  • mariadb-connector-j/src/main/java/org/mariadb/jdbc/internal/failover/impl/AuroraListener.java
  • mariadb-connector-j/src/main/java/org/mariadb/jdbc/internal/protocol/AuroraProtocol.java

https://github.com/MariaDB/mariadb-connector-j/blob/master/documentation/failover-and-high-availability-with-mariadb-connector-j.creole

1
javamysqljdbcmariadbamazon-aurora

1 Answers

2
votes

Here are some hints : Aurora has many instances. One is "writer" (Master), others are "Reader"(Slaves).

When a writer is down, one slave will be promoted new master, other slaves will now replicate from this new master (automatic reboot). If the old master come up again, it will become a slave.

Aurora has a DNS endpoint for cluster like "xx.cluster-yy.zz.rds.amazonaws.com" that point to current master. When a failover occur, DNS is refreshed ... but not immediatly.

A "connection" to aurora mean 2 underlying connection to instances: one to master, one to slave. Driver will use the underlying connection to master or slave according to Connection.setReadonly().

Every time driver connect to an instance, it will ensure current state checking global variable "innodb_read_only" (OFF = master).

Aurora instances can be added, so on initial connection, using user cluster endpoint, the current list of instances will be retrieved using information_schema.replica_host_status.

To established the 2 underlying connections, driver will connect to a random host, if this is the current master, good then all other hosts are slaves, if not, driver will ask the slave his current master so next connection will connect host using information_schema.replica_host_status where session_id = 'MASTER_SESSION_ID' (more relyable than using DNS). If connection to an instance fail, this instance name will be put in a blacklist for a certain amount of time (this blacklist is shared per jvm) to avoid reusing it. Driver try to reconnect a random available host until there is none not blacklisted, then can retry with blacklisted one for some time (depending on parameters). If connection is successful, then instance is "un-blacklisted".

For failover of underlying slave connection, master connection is then used, and some underlying pool of thread will then try to reconnect a slave instance in background.