Datastax Enterprise: Error during nodetool cleanup

0
votes

I've got a two-datacenter deployment with cassandra and search nodes. It looks like this:

ubuntu@ip-172-31-25-223:~$ dsetool ring
Note: Ownership information does not include topology, please specify a keyspace. 
Address          DC           Rack         Workload         Status  State    Load             Owns                 VNodes                                      
172.31.47.194    Solr         2a           Unknown          Up      Normal   3.17 GB          0.00%                1                                           
172.31.39.59     us-west-2    2a           Unknown          Up      Normal   2.32 GB          31.53%               512                                         
172.31.9.36      us-west-2    2c           Unknown          Up      Normal   3.43 GB          33.48%               512                                         
172.31.25.223    us-west-2    2b           Unknown          Up      Normal   3.25 GB          34.99%               512                                         
Warning:  Node 172.31.25.223 is serving 1.11 times the token space of node 172.31.39.59, which means it will be using 1.11 times more disk space and network bandwidth. If this is unintentional, check out http://wiki.apache.org/cassandra/Operations#Ring_management

I recently added the 172.31.39.59 node and I wanted to run cleanups on the other nodes to remove some of the duplicate data.

Running nodetool cleanup on the 172.31.25.223 node resulted in the following trace:

ubuntu@ip-172-31-25-223:~$ nodetool cleanup
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IllegalStateException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:228)
        at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
        at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2251)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException
        at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex.delete(Cql3SolrSecondaryIndex.java:61)
        at org.apache.cassandra.db.index.SecondaryIndexManager.deleteFromIndexes(SecondaryIndexManager.java:470)
        at org.apache.cassandra.db.compaction.CompactionManager$CleanupStrategy$Full.cleanup(CompactionManager.java:720)
        at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:580)
        at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:275)
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:223)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more
Caused by: java.lang.NullPointerException
        at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doDelete(AbstractSolrSecondaryIndex.java:619)
        at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex.delete(Cql3SolrSecondaryIndex.java:57)
        ... 10 more

I'm new to Datastax enterprise and I don't have much visibility into Cassandra->Solr and how the pipeline works. I'd appreciate any help!

Thanks!

Advait

Update: Tried running it again and got a different trace this time:

ubuntu@ip-172-31-25-223:~$ nodetool cleanup
Exception in thread "main" java.lang.AssertionError: [SSTableReader(path='/raid0/cassandra/data/liminex_ent/sub_accounts/liminex_ent-sub_accounts-jb-73-Data.db'), SSTableReader(path='/raid0/cassandra/data/liminex_ent/sub_accounts/liminex_ent-sub_accounts-jb-74-Data.db')]
        at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2130)
        at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2127)
        at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2109)
        at org.apache.cassandra.db.ColumnFamilyStore.markAllCompacting(ColumnFamilyStore.java:2140)
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:215)
        at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
        at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2251)
        at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
1
datastax-enterprisedatastax
What version of DSE are you running? - dkblinux98
@dkblinux98 I'm running DSE 4.5.1 - advait

1 Answers

1
votes

This was a bug - fixed in 4.5.3

The fix for the Cleanup bug is in our latest release 4.5.3 which is GA yesterday, Nov 4th 2014.

The rolling upgrade should be straight forward/no downtime.

Here's the release notes:

http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/RNdse45.html

Fixed an issue causing a null pointer exception on non Solr workload nodes holding Solr data and attempting to run the nodetool cleanup command on data. (DSP-4310)