org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId

1
votes

In my live cassandra cluster, I have accidentally dropped the keyspace. Using snapshots, I have recovered the data but now the response time is very high, though cassandra recentReadLatencyMicros in < 2ms on all nodes.

After restore, I am getting following exception very frequently, I have created all the column families again but still getting the exception. How do I know by cfId which column family I am missing. I had also checked in schema_columnfamilies but this cfId doesn't exist. Any help is greatly appreciated.

ERROR [RequestResponseStage:1094556] 2014-04-01 03:12:05,583 AbstractCassandraDaemon.java (line 132) Exception in thread Thread[RequestResponseStage:1094556,5,main] java.io.IOError: org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1118 at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71) at org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1118 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126) at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:72) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:109) at org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:81) at org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)

1
cassandra

1 Answers

0
votes

I. in CQLSH run this to get the list of CFs in your restored (broken) keyspace:

SELECT columnfamily_name FROM system.schema_columnfamilies 
WHERE keyspace_name='your_keyspace';

(Replace your_keyspace with an appropriate name.)

II. Go to your snapshot and see what directories there are.

III. Check if any items in the second list are missing from the first. If you have hundreds of CFs, you may want to use some scripting to quickly find the missing table. Or just paste both lists into a spreadsheet and sort. Then by eyeballing you should be able to quickly see the mismatched row.