Not able to start cassandra

Question

I am facing issue with Cassandra, whenever I try to start Cassandra I am getting Too many files open error.

I have increased the file descriptor to 1000000, still the same error.

[UPDATED]

I went through the debug logs, on start it is opening many sstables. Here are the logs

DEBUG [SSTableBatchOpen:2] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-159921-big (60 bytes) DEBUG [SSTableBatchOpen:1] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-166961-big (49 bytes) DEBUG [SSTableBatchOpen:4] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-181150-big (57 bytes) DEBUG [SSTableBatchOpen:3] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-188190-big (49 bytes) DEBUG [SSTableBatchOpen:2] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-173741-big (54 bytes) DEBUG [SSTableBatchOpen:1] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-159987-big (45 bytes) DEBUG [SSTableBatchOpen:3] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-167027-big (49 bytes) DEBUG [SSTableBatchOpen:4] 2017-06-20 11:03:40,635 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-198208-big (53 bytes) DEBUG [SSTableBatchOpen:1] 2017-06-20 11:03:40,636 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-177001-big (48 bytes) DEBUG [SSTableBatchOpen:2] 2017-06-20 11:03:40,636 SSTableReader.java:479 - Opening /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-184041-big (57 bytes Here are the system logs:

ERROR [SSTableBatchOpen:1] 2017-06-19 19:08:40,175 CassandraDaemon.java:205 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:127) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:91) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:125) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:745) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:706) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:492) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:375) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)
        at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_101]
        at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_101]
        at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_101]
        at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[na:1.8.0_101]
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:100) ~[apache-cassandra-3.0.9.jar:3.0.9]
        ... 15 common frames omitted
ERROR [SSTableBatchOpen:1] 2017-06-19 19:08:40,177 JVMStabilityInspector.java:140 - JVM state determined to be unstable.  Exiting forcefully due to:
java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)

Check the existence of this file /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db — Ashraful Islam
@Arvind seems to be java error rather than cassendra : 1. Try restarting your box, to free space. 2. If, not following 1.try kill your local java threads and restart your IDE 3. stackoverflow.com/questions/13706409/… checkout the link for updating ulimit. I have faced the same error for me restarting worked fine. — anshul Gupta
Check this docs.datastax.com/en/archived/cassandra/2.0/cassandra/… — Ashraful Islam
15GB space is available and I have tried restarting the machine, didn't work for me. I have already increased the Ulimit. — Arvind

Mandraenke Mandraenke · Accepted Answer · 2017-06-20T08:04:04

As I cant comment on the previous answer, here a small hint:

How is your cassandra started? How is it installed? It's possible that your ulimit changes are not affecting the user cassandra runs on (double check with ls -l in your data directories to see who ones the files). With debian packages cassandra runs as user cassandra and limits are set as

cassandra01:/etc$ cat security/limits.d/cassandra.conf
# Provided by the cassandra package
cassandra  -  memlock  unlimited
cassandra  -  nofile   100000
cassandra  -  as       unlimited
cassandra  -  nproc    8096
cassandra01:/etc$

How many sstables do you have in your data directory?

Try to find out how many files are open util the crash with something like this:

lsof -n | grep java

Not able to start cassandra

1 Answers