I am deploying clickhouse along with my Sentry On premise setup hosted in AWS EKS. I want to do volume management of clickhouse data /var/lib/clickhouse using a PVC using Storage class (EBS). I want to have a single block device to which all the clickhouse pods can write and read from. I am unable to achieve this using kubernetes kind: deployment with 2 replicas. The first pod gets the volume and starts running without any error while the second pod keeps crashing with the following message:
kubelet[3862]: E1210 10:03:17.377448 3862 pod_workers.go:191] Error syncing pod ab24c325-a86c-4fa4-aee4-d14766d7cfec ("clickhouse-59cbd46586-vsmvg_default(ab24c325-a86c-4fa4-aee4-d14766d7cfec)"), skipping: failed to "StartContainer" for "clickhouse-server" with CrashLoopBackOff: "back-off 5m0s restarting failed container=clickhouse-server pod=clickhouse-59cbd46586-vsmvg_default(ab24c325-a86c-4fa4-aee4-d14766d7cfec)"
Using clickhouse helm chart which is available as stateful set I have running containers. My question is: If clickhouse is running as stateful set with 2 or more replicas, the data across each pod's volume will be in-consistent and how that is managed? Is there a way to achieve this with single volume with multiple pods talking to it?
