Apache Ignite Off Heap memory filling up

Question

We have the following ignite cluster setup configuration:

Apache Ignite version : 2.7.5
Ignite persistence is enabled (true)
2 node cluster in partitioned mode
RAM - 210 GB per node
JVM xms and xmx 20G
Off Heap Memory Max: 120GB
Number of records - 160 million

I can see the following node metrics:

[03:13:31,126][INFO][db-checkpoint-thread-#146%GridA%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=df22db5b-6ffa-4f5d-b6da-d0e36c0492af, pages=1512, markPos=FileWALPointer [idx=6659, fileOff=249851578, len=49197], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=26ms, pagesWrite=13ms, fsync=312ms, total=351ms]
[03:14:05,346][INFO][grid-timeout-worker-#67%GridA%][IgniteKernal%GridA] 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=25a3a57c, name=GridA, uptime=16 days, 22:40:01.512]
    ^-- H/N/C [hosts=10, nodes=10, CPUs=172]
    ^-- CPU [cur=1.17%, avg=4.94%, GC=0%]
    ^-- PageMemory [pages=30333907]
    ^-- Heap [used=3889MB, free=81.01%, comm=20480MB]
    ^-- Off-heap [used=119880MB, free=2.68%, comm=123179MB]
    ^--   sysMemPlc region [used=0MB, free=99.99%, comm=99MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.82%, comm=99MB]
    ^--   Default_Region region [used=119880MB, free=2.44%, comm=122880MB]
    ^--   TxLog region [used=0MB, free=100%, comm=99MB]
    ^-- Ignite persistence [used=253233MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   metastoreMemPlc region [used=unknown]
    ^--   Default_Region region [used=253233MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]

Does the ignite node require restart or should page replacement trigger and free up some offheap space?

Edit-2: as you can see that off heap memory free space is ~ 2.5 % and still page replacement(PR) hasn't been triggered. Could not find anything on the topic as to when PR will be triggered. Will it be triggered at free space = 0% ? Is there a possibility that my ignite node would shutdown if free space reaches 0%? Any implications on query performance when page replacement triggers eventually?

Pavel Vinokurov Pavel Vinokurov · Accepted Answer · 2020-12-07T07:35:22

In case of enabled persistence after a data region is filled up Page replacement is triggered.

Apache Ignite Off Heap memory filling up

1 Answers