Solr - lost configuration after recovering Zookeeper

Question

I just inherited a hadoop cluster ( never worked with hadoop before ) consisting of 7 servers and administered through Ambari.
Today Ambari lost heartbeat with all services on server3 as well as ZooKeeper services (hosted on servers 1, 2, and 3), ZKFailOver ( hosted on server 1 and 2 ), and ZooKeeper clients ( hosted on 4,5,6,7 ) stopped and all refused to start. This also caused the Solr services to stop working.
After some investigating I found that Zookeeper on server3 was erroring on recent snapshot due to a CRC problem. After some more reading I removed the old snapshot files in .../zookeeper/version-2/ and ran 'zk -formatZK' (on server1). Zookeeper services are now able to start and heartbeat from server3 are being received.
The problem I see now is all the Solr services are no longer configured properly - "...ZooKeeperException: Could not find configName for collectioin xxxx found:null" I haven't been having much success figuring out how to get the previous Solr configurations to Zookeeper. I'm trying to use 'zkcli.sh' that I found in the Solr directory which is located in '/opt/solr/xxxx/scripts/cloud-scripts/' but it doesn't seem to work like the zkCli described in the Hadoop documentation. My question is, how do I setup the Solr servers using the existing config files? If I can't how can I go about reconstructing the following configuration:

                            / --- server5
                 /--shard1------- server7
        core -- <
                 \--shard2------- server4
                            \ --- server6

Thanks.

Questions on professional server- or networking-related infrastructure administration are off-topic for Stack Overflow unless they directly involve programming or programming tools. You may be able to get help on Server Fault. — Gustavo Morales

Mike_ Mike_ · Accepted Answer · 2016-06-17T20:48:09

So after trial and error I found that zkcli.sh should be used in the following manner:

./zkcli.sh server1:2181,server2:2181,server3:2181 -cmd upconfig .../solr/<corename>/conf -confname <configfilename>

This should upload any existing config to all ZK nodes.

Solr - lost configuration after recovering Zookeeper

1 Answers