java.lang.OutOfMemoryError: Java heap space on ignite cluster

0
votes

We are migrating a web application from an ad hoc in memory cache solution to an apache ignite cluster, where the jboss that runs the webapp works as client node and two external vm works as ignite server nodes.

When testing performance with one client node and one server node all goes ok. But when testing with one client node and two server nodes in cluster, the server nodes crash with an OutOfMemoryError.

The virtual machine of both nodes it's started with -server -Xms1024M -Xmx1024M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:NewSize=128m -XX:MaxNewSize=128m -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:MaxGCPauseMillis=1000 -XX:InitiatingHeapOccupancyPercent=50 -XX:+UseCompressedOops -XX:ParallelGCThreads=8 -XX:ConcGCThreads=8 -XX:+DisableExplicitGC

Any idea why a two nodes cluster fails when a single node one works perfectly running the same test ?

I don't know if it's relevant, but the test consists on 10 parallel http requests launched against the JBoss server, that each one starts a process that writes several entries into the caché.

2
ignite

2 Answers

0
votes

The communication between nodes can add some overhead, so apparently 1GB is not enough for the data and Ignite itself. Generally 1GB is not enough, I would recommend to allocate at least 2GB, better 4GB per node.

0
votes

In the end the problem wasn't in the amount of memory required by the two nodes, but in the synchronization between the nodes. My test caché was running with PRIMARY_SYNC, but write/read cycles where faster than the replication in the cluster and end up in an inconsistent read that provoked an infinite loop that wrote infinite values to the cluster.

Changing to FULL_SYNC fixed the problem.