I'm trying to implement a kafka cluster consists of three nodes of kafka servers and three nodes of zookeeper servers using docker containers, which of the following is preferred or if neither, what is the preferred way?
- three docker containers each hosting a kafka/zookeeper server pair
- six docker containers with three of them for kafka servers and three others for zookeeper servers
I'm asking this because it seems to me like a three-node Zookeeper cluster only survives single-node failure, whilst a three-node Kafka cluster could potentially survive a two-node failure (you may have to set the topic replication factor to 3). So is it better to run them in different containers if it isn't so costly to create new containers? Speaking of which, how costly is it to start a new Docker container?
In case I am advised to run one server per container, is it more preferred to build a tailored Docker image for every kind of server (in this case, one docker image for kafka and another for zookeeper), or one unified image for all different servers? I'm guessing it doesn't make sense to create two separate images just for kafka and zookeeper but what if i have all different kinds of clusters and servers, think elasticsearch, to simulate? at what point would it start to make sense to create different docker images to be used inside a single project?
