Solr schema "Already closed" and related errors after base DSE/Solr Node setup

Question

Getting started with DSE Solr nodes, initial setup fine and was able to follow this example with no issues:

http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/srch/srchTutStrt.html

My first test use case is some example location data, modifying the tutorial example. I am now at a state where I can create my table, insert ~5K example rows, and when pushing the schema get the following exception:

<?xml version="1.0" encoding="UTF-8"?>
  <response>
    <lst name="responseHeader"><int name="status">500</int><int name="QTime">245</int></lst><lst name="error"><str name="msg">Already closed</str><str name="trace">org.apache.solr.common.SolrException: Already closed
    at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:851)
    at com.datastax.bdp.search.solr.core.CassandraCoreContainer.doReload(CassandraCoreContainer.java:700)
    at com.datastax.bdp.search.solr.core.CassandraCoreContainer.create(CassandraCoreContainer.java:224)
    at com.datastax.bdp.search.solr.core.SolrCoreResourceManager.createCore(SolrCoreResourceManager.java:256)
    at com.datastax.bdp.search.solr.handler.admin.CassandraCoreAdminHandler.handleCreateAction(CassandraCoreAdminHandler.java:117)
    at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137)
    at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:669)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:99)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:891)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:750)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.lucene.store.AlreadyClosedException: Already closed at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340) at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:262) at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:480) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:772) ... 33 more 500name=snet_data.location_test1&action=CREATE

Using this as my table create statement:

CREATE table location_test1 (
"id" TIMEUUID,
"source_id" UUID,
"name" VARCHAR,
"address" VARCHAR,
"address_extended" VARCHAR,
"po_box" VARCHAR,
"locality" VARCHAR,
"region" VARCHAR,
"post_town" VARCHAR,
"admin_region" VARCHAR,
"postcode" VARCHAR,
"country" VARCHAR,
"tel" VARCHAR,
"latlon" VARCHAR,
"neighborhood" SET<VARCHAR>,
"website" VARCHAR,
"email" VARCHAR,
"category_ids" SET<VARCHAR>,
"status" VARCHAR,
"chain_name" VARCHAR,
"chain_id" UUID,
PRIMARY KEY ("id"));

With the solr schema:

<schema name="location_test1" version="1.5">
 <types>
  <fieldType name="string" class="solr.StrField"/>
  <fieldType name="text" class="solr.TextField">
    <analyzer>
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
  </fieldType>
  <fieldType name="geo" class="solr.GeoHashField"/>
  <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0" />
  <fieldType name="int" class="solr.TrieIntField"/>
  <fieldType name="uuid" class="solr.UUIDField"/>
 </types>
 <fields>
   <field name="id" type="uuid" indexed="true"  stored="true" docValues="true"/>
   <field name="name" type="string" indexed="true"  stored="true"/>
   <field name="latlon" type="geo" indexed="true"  stored="true"/>
</fields>
<defaultSearchField>name</defaultSearchField>
<uniqueKey>(id)</uniqueKey>
</schema>

UPDATED (10/29) after new tests

So after seeming like these errors are being caused b/c DSE Solr is in some bad state, even after dropping table and data and starting over, I decided to drop the entire keyspace as the restart point. Getting different behavior now.., consistent with earlier errors where on core creation it complains that a multi-value field should be mapped to List/Set type.

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">500</int><int name="QTime">325</int></lst><lst name="error"><str name="msg">Unable to create core: snet_data.location_test1</str><str name="trace">org.apache.solr.common.SolrException: Unable to create core: snet_data.location_test1
        at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:957)
        at com.datastax.bdp.search.solr.core.CassandraCoreContainer.create(CassandraCoreContainer.java:266)
        at com.datastax.bdp.search.solr.core.SolrCoreResourceManager.createCore(SolrCoreResourceManager.java:256)
        at com.datastax.bdp.search.solr.handler.admin.CassandraCoreAdminHandler.handleCreateAction(CassandraCoreAdminHandler.java:117)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137)
        at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:669)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
        at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:99)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:891)
        at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:750)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2283)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Multi-valued field status should be mapped to either List or Set types, found: org.apache.cassandra.db.marshal.UTF8Type
        at com.datastax.bdp.search.solr.core.Cql3CassandraSolrSchemaUpdater.update(Cql3CassandraSolrSchemaUpdater.java:115)
        at com.datastax.bdp.search.solr.core.CassandraCoreContainer.create(CassandraCoreContainer.java:245)
        ... 31 more
</str><int name="code">500</int></lst><str name="params">name=snet_data.location_test1&amp;action=CREATE</str>
</response>

Just like before with other field errors, the status field it's complaining about is defined in the table as a varchar, and in the schema as a string, so not quite sure why it complains about these.

What I have done now is stripped the schema down to just id,name,latlon. Back to where I don't get the multi-value errors on single value varchar/string fields.., back to original "Already Closed" error

Here are my curl statements, built from example referenced above from datastax solr tutorial:

curl http://10.0.1.212:8983/solr/resource/snet_data.location_test1/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'

curl http://10.0.1.212:8983/solr/resource/snet_data.location_test1/schema.xml --data-binary @schema.xml -H 'Content-type:text/xml; charset=utf-8'

curl "http://10.0.1.212:8983/solr/admin/cores?action=CREATE&name=snet_data.location_test1"

Steps Taken in running setup tests:

Log into cql shell and do following

Create keyspace:

CREATE KEYSPACE snet_data WITH REPLICATION =
       {'class':'NetworkTopologyStrategy', 'Solr':1};

Create table:

CREATE table location_test1 (
"id" TIMEUUID,
"source_id" UUID,
"name" VARCHAR,
"address" VARCHAR,
"address_extended" VARCHAR,
"po_box" VARCHAR,
"locality" VARCHAR,
"region" VARCHAR,
"post_town" VARCHAR,
"admin_region" VARCHAR,
"postcode" VARCHAR,
"country" VARCHAR,
"tel" VARCHAR,
"latlon" VARCHAR,
"neighborhood" SET<VARCHAR>,
"website" VARCHAR,
"email" VARCHAR,
"category_ids" SET<VARCHAR>,
"status" VARCHAR,
"chain_name" VARCHAR,
"chain_id" UUID,
PRIMARY KEY ("id"));

(tried both importing test of 5k records like tutorial, also running solr curl commands sans inserting initial data)

Run solr curl commands to setup config, schema, core:

curl http://10.0.1.212:8983/solr/resource/snet_data.location_test1/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'

curl http://10.0.1.212:8983/solr/resource/snet_data.location_test1/schema.xml --data-binary @schema.xml -H 'Content-type:text/xml; charset=utf-8'

curl "http://10.0.1.212:8983/solr/admin/cores?action=CREATE&name=snet_data.location_test1"

Answer 1

I could reproduce your error after using your schema and create table statements. After crashing a few times trying to create the core the server finally gives that error, probably because it is in an inconsistent state.

If you use the following in your schema I guess all will be ok for you. I would need to know exact versions to be 100% sure though.

<?xml version="1.0" encoding="UTF-8" ?>
<schema name="myTest" version="1.5">

To avoid future problems here is what I would recommend you:

• Look for the 'wiki' demo in the demos folder.
• Use that as a template. Change scripts' core names, change the schema, change the solrconfig, etc.
• This will be convenient as the scripts do the work for you, you have already working schemas etc you can modify and be sure they work

Hope it helps.

UPDATE 10/30:

I would need your exact version but you've made progress as at one point I did also get that 'status' multivalued field error. Let's go step by step:

• Stop the server and delete stuff. In my case: rm -rf /var/lib/cassandra/*
• Start the server: dse cassandra -s

• Create the ks and table with the cqlsh shell:

  CREATE KEYSPACE wiki WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

  USE wiki;

  CREATE table location_test1 ( "id" TIMEUUID, "source_id" UUID, "name" VARCHAR, "address" VARCHAR, "address_extended" VARCHAR, "po_box" VARCHAR, "locality" VARCHAR, "region" VARCHAR, "post_town" VARCHAR, "admin_region" VARCHAR, "postcode" VARCHAR, "country" VARCHAR, "tel" VARCHAR, "latlon" VARCHAR, "neighborhood" SET, "website" VARCHAR, "email" VARCHAR, "category_ids" SET, "status" VARCHAR, "chain_name" VARCHAR, "chain_id" UUID, PRIMARY KEY ("id"));

• Edit schema.xml and copy your own. I copied the one in your question. I use the solrconfig from the wiki demo.

curl http://:8983/solr/resource/wiki.location_test1/schema.xml --data-binary @schema.xml -H 'Content-type:text/xml; charset=utf-8'

SUCCESS

curl http://<host>:8983/solr/resource/wiki.location_test1/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'

SUCCESS

curl "http://<host>:8983/solr/admin/cores?action=CREATE&name=wiki.location_test1"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1598</int></lst>
</response>

It should work. If it doesn't I need your exact versions. Sorry about the formatting, I can't get it to show as code...