java.lang.OutOfMemoryError: Direct buffer memory and Not marking nodes down due to local pause of

1
votes

one of my cassandra node is running on an aws instance, some time it is getting stopped, I have seen the following info in the system.log and cassandra.log files. What might be the reasons for these kind of errors

java.lang.OutOfMemoryError: Direct buffer memory

and 

Not marking nodes down due to local pause of

How to solve these issues

1
javacassandra
Did you try to produce heapdump on OOME and analyze it? Also, which version of Cassandra do you use? – artie
Cassandra version is 3.0.9 and how to produceheapdump on OOME ? – Sat
Check this documentation: docs.datastax.com/en/cassandra/3.0/cassandra/configuration/… – Alex Ott

1 Answers

2
votes

The solution was installing Java 8 on all of the nodes and changing the GC strategy to G1, and the results were amazing. I’ve launched all nodes with 8GB of heap with the G1 Garbage collection algorithm and I saw an immediate improvement, the heap used didn’t go past 4GB per node, and the GC times decreased.

I’ll explain the steps I took to change the setting to run with the G1GC and I’ll elaborate a bit on how everything works.

(After I finished writing this post, I saw that it was just too long, so I’ve decided to be nice and left A lot of things for the enrichment parts, so feel free to skip them)

Install the newest Oracle JDK 8 to have the latest update of the G1GC. This will also work with Java JRE / JDK later than Java 7 update 4, but in the manner of Apache Cassandra Java 8 is recommended.

Installing Java 8 (For Debian based systems): Adding the Oracle repository, updating the local apt-get cache and installing oracle-java8-installer package:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Automated installation (auto accept license)

echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections

sudo apt-get install oracle-java8-set-default

Setting “JAVA_HOME” variable in the environment:

sudo vim /etc/environment

Add these lines in the end of the file.

 JAVA_HOME="/usr/lib/jvm/java-8-oracle/"
 PATH=$PATH:$JAVA_HOME/bin

Reload the newly added environment variables:

source /etc/environment

We now have the Java 8 JDK installed.

In order to setup the G1 algorithm to run on a specific node you’ll need to follow these steps:

  1. In the cassandra-env.sh file add the follwing lines:

    G1 - Parameters

    JVM_OPTS="$JVM_OPTS -XX:+UseG1GC" JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5" JVM_OPTS="$JVM_OPTS -XX:+PrintFlagsFinal"

  2. You’ll need to comment out all of the CMS specific flags that will probably be there by default. search for these and be careful for others:

    -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSClassUnloadingEnabled

And that’s it,

just restart the node and you’ll be up and running!

Reference:- http://progexc.blogspot.in/2015/12/taking-care-of-garbage-in-cassandra.html