Issue while creating Solr core on HDFS

Question

I am trying to create solr core on HDFS in a stand alone instance (solr-5.3.0 and Hadoop 2.7). I have started the service like below,

$ bin/solr start -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs -Dsolr.data.dir=hdfs://localhost:9000/tmp -Dsolr.updatelog=hdfs://localhost:9000/tmp -s solr-cores/core1

Waiting up to 30 seconds to see Solr running on port 8983 [/] Started Solr server on port 8983 (pid=42277). Happy searching!

And trying to create core like below,

bin/solr create -c hdfsstarted -d /home/admin/HadoopTools/solr-5.3.0/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n hdfsstarted

But getting below error:

Setup new core instance directory: /home/admin/HadoopTools/solr-5.3.0/solr-cores/core1/hdfsstarted

Creating new core 'hdfsstarted' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=hdfsstarted&instanceDir=hdfsstarted

ERROR: Error CREATEing SolrCore 'hdfsstarted': Unable to create core [hdfsstarted] Caused by: Protocol message end-group tag did not match expected tag.

I have modified the solrconfig.xml like below,

<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
    <str name="solr.hdfs.home">hdfs://10.67.5.244:50070/tmp</str>
    <bool name="solr.hdfs.blockcache.enabled">true</bool>
    <int name="solr.hdfs.blockcache.slab.count">1</int>
    <bool name="solr.hdfs.blockcache.direct.memory.allocation">false</bool>
    <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
    <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
    <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
    <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
    <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
    <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
<lockType>
   hdfs
</lockType>

Kindly let me know how to create core correctly in HDFS.

Were you able to resolve this? Seeing same error with a Solr->HDFS config using solr 5.2.1 and hdp 2.4. — bug11

arghtype arghtype · Accepted Answer · 2018-11-21T19:12:01

Caused by: Protocol message end-group tag did not match expected tag.

This error happens because you are using incorrect HDFS port. See hdfs - ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: .

Here you need to change port from 50070 (which looks like NameNode web ui port) to 8020 or whatever port you are using as NameNode RPC port:

<str name="solr.hdfs.home">hdfs://10.67.5.244:50070/tmp</str>

Issue while creating Solr core on HDFS

1 Answers