Re-add lost Clickhouse replica in Zookeeper cluster

1
votes

We previously had three Clickhouse nodes perfectly synced within Zookeeper until one of them was lost.

The Clickhouse node was rebuilt exactly as it was before (with Ansible) and the same create table command was run which resulted in the following error.

Command:

CREATE TABLE ontime_replica ( ... )
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/ontime_replica', '{replica}', FlightDate, (Year, FlightDate), 8192)

The error is:

Received exception from server:
Code: 253. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: Replica /clickhouse/tables/01/ontime_replica/replicas/clickhouse1 already exists..

We're currently using Zookeeper version 3.4.10 and I would like to know if there's a way to remove the existing replica within Zookeeper, or simple let Zookeeper know that this is the new version of the existing replica.

Thank you in advance!

1
apache-zookeeperclickhouse

1 Answers

3
votes

My approach to the solution was incorrect. Originally, I thought I needed to remove the replica within Zookeeper. Instead, the following commands within the Clickhouse server solve this problem.

  • Copy the SQL file from another, working node. The file is in /var/lib/clickhouse/metadata/default

  • chown clickhouse:clickhouse <database>.sql

  • chmod 0640 <database>.sql

  • sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data

  • service clickhouse-server start