Clickhouse Replication not happening 'DB::Exception: No interserver IO endpoint named'

Question

Following is the error in system.replication_queue table:

Code: 86, e.displayText() = DB::Exception: Received error from remote server ?endpoint=DataPartsExchange%3A%2Fclickhouse%2Ftables%2Freplicated%2Ftest8%2Freplicas%2FR01&part=1009_0_0_0&client_protocol_version=2&compress=false. HTTP status code: 500 Internal Server Error, body: Code: 221, e.displayText() = DB::Exception: No interserver IO endpoint named DataPartsExchange:/clickhouse/tables/replicated/test8/replicas/R01, Stack trace (when copying this message, always include the lines below):

I've 3 zookeeper nodes, and following is the config on all 3:

    <zookeeper incl="zookeeper" optional="true" />
        
            <node index="1">
            <host>ZK_Node1/host>
            <port>2181</port>
            </node>
    
            <node index="2">
            <host>ZK_Node2</host>
            <port>2181</port>
            </node>
    
            <node index="3">
            <host>ZK_Node3</host>
            <port>2181</port>
            </node>    
</zookeeper>

And its a 2 node replica set, so following are settings on each of the replica:

        <interserver_http_host>HostName of server</interserver_http_host> 

<remote_servers incl="clickhouse_remote_servers" >        
                <Prod_Cluster>
                    <shard>
                        <replica>
                            <host>Rep1_IPAddress</host>
                            <port>9000</port>
                        </replica>
                       <replica>
                             <host>Rep2_IPAddress</host>
                             <port>9000</port>
                         </replica>
                     </shard>
                <Prod_Cluster>
            </remote_servers>
        
    <macros incl="macros" optional="true" />
                <cluster>Prod_Cluster</cluster>
                <shard>1</shard>
                <replica>R01</replica>
            </macros>

Not sure, what am I missing to avoid the mentioned error.

is it ok that config-files are not valid XML? Look for example - <zookeeper incl="zookeeper" optional="true" /> <node index="1"> <host>ZK_Node1/host>. — vladimir
just check: you provided <macros> just for one node; did you define the separate macros-config for the second node (that should have another value in replica-field)? — vladimir

vladimir vladimir · Accepted Answer · 2020-09-11T18:40:24

Make sure that port 9009 is not blocked, it used for inter-server replication.

Recommended to turn on internal_replication (see details here)

<remote_servers incl="clickhouse_remote_servers" >        
  <Prod_Cluster>
    <internal_replication>true</internal_replication> 
    <shard>
      <replica>
      ..

Clickhouse Replication not happening 'DB::Exception: No interserver IO endpoint named'

2 Answers