Unable to start DSE using SPARK_ENABLED=1

1
votes

We are running 6 node cluster with:

HADOOP_ENABLED=0
SOLR_ENABLED=0
SPARK_ENABLED=0
CFS_ENABLED=0

Now, we would like to add Spark to all of them. It seems like "adding" is not the right term because this would not fail. Anyways, the steps we've done: 1. drained one of the nodes 2. changed /etc/default/dse to SPARK_ENABLED=1 and HADOOP_ENABLED=0 3. sudo service dse restart

And got the following in the log:

ERROR [main] 2016-05-17 11:51:12,739 CassandraDaemon.java:294 - Fatal exception during initialization org.apache.cassandra.exceptions.ConfigurationException: Cannot start node if snitch's data center (Analytics) differs from previous data center (Cassandra). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.

There are two related questions that have been already answered:

  1. Unable to start solr aspect of DSE search
  2. Two node DSE spark cluster error setting up second node. Why?

Unfortunately, clearing the data on the node is not an option - why would I do that? I need the data to be intact.

Using "-Dcassandra.ignore_rack=true -Dcassandra.ignore_dc=true" is a bit scary in production. I don't understand why DSE wants to create another DC and why can't it just use the existing one?

I know that according to datastax's doc one should partition the load using different DC for different workloads. In our case we just want to run SPARK jobs on the same nodes that Cassandra is running using the same DC.

Is that possible?

Thanks!

1
apache-sparkcassandradatastaxdatastax-enterprise

1 Answers

1
votes

The other answers are correct. The issue here is trying to warn you that you have previously identified this node as being in another DC. This means that it probably doesn't have the right data for any key-spaces with Network Topology Strategy. For example if you had a NTS keyspace which had only one replica in "Cassandra" and changed the DC to "Analytics" you could inadvertently lose all of the data.

This warning and the accompanying flag are telling you that you are doing something that you should not be doing in a production cluster.

The real solution to this is to explicitly name your dc's using GossipingFileSnitch and not rely on SimpleSnitch which names based on the DSE workload.

In this case, switch to GPFS and set the DC name to Cassandra.