Getting out of memory on Ignite server node

Question

I have setup an ignite 2.3 server node along with 32 clients nodes. After running the multiple query, I have been observed Out Of Memory Error in server node logs.

Server Configuration:

Configure 4 GB java max heap memory.
Ignite Persistence is disabled
Using default data region.
Using Spring Data for apply the query on ignite node.

Captured memory snapshots of ignite server node.

- Node [id= =44:33:12.948]
    ^-- H/N/C [hosts=32, nodes=32, CPU =39]
    ^-- CPU [cur=3.7%, avg=0.23%, G C =0%]
    ^-- Page Memory [pages=303325]
    ^-- Heap [used=2404 MB, free=36.21%, comm=3769 MB]
    ^-- Non heap [used=78 MB, free=-1%, comm=80 MB]
    ^-- Public thread pool [active=0, idle=0, q Size =0]
    ^-- System thread pool [active=0, idle=6, q Size=0]
    ^-- Outbound messages queue [size=0]

Heap Dump Logs Analysis :

query-#8779
  at java.nio.Bits$1.newDirectByteBuffer(JILjava/lang/Object;)Ljava/nio/ByteBuffer; (Bits.java:758)
  at org.apache.ignite.internal.util.GridUnsafe.wrapPointer(JI)Ljava/nio/ByteBuffer; (GridUnsafe.java:113)
  at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.pageBuffer(J)Ljava/nio/ByteBuffer; (PageMemoryNoStoreImpl.java:253)
  at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(Lorg/apache/ignite/internal/processors/cache/CacheGroupContext;Lorg/apache/ignite/internal/processors/cache/GridCacheSharedContext;Lorg/apache/ignite/internal/pagemem/PageMemory;Lorg/apache/ignite/internal/processors/cache/persistence/CacheDataRowAdapter$RowData;)V (CacheDataRowAdapter.java:167)
  at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(Lorg/apache/ignite/internal/processors/cache/CacheGroupContext;Lorg/apache/ignite/internal/processors/cache/persistence/CacheDataRowAdapter$RowData;)V (CacheDataRowAdapter.java:102)
  at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(J)Lorg/apache/ignite/internal/processors/query/h2/opt/GridH2Row; (H2RowFactory.java:62)
  at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasLeafIO.getLookupRow(Lorg/apache/ignite/internal/processors/cache/persistence/tree/BPlusTree;JI)Lorg/h2/result/SearchRow; (H2ExtrasLeafIO.java:126)
  at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasLeafIO.getLookupRow(Lorg/apache/ignite/internal/processors/cache/persistence/tree/BPlusTree;JI)Ljava/lang/Object; (H2ExtrasLeafIO.java:36)
  at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(Lorg/apache/ignite/internal/processors/cache/persistence/tree/io/BPlusIO;JILjava/lang/Object;)Lorg/apache/ignite/internal/processors/query/h2/opt/GridH2Row; (H2Tree.java:123)
  at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(Lorg/apache/ignite/internal/processors/cache/persistence/tree/io/BPlusIO;JILjava/lang/Object;)Ljava/lang/Object; (H2Tree.java:40)
  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(JLorg/apache/ignite/internal/processors/cache/persistence/tree/io/BPlusIO;II)Z (BPlusTree.java:4548)
  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.nextPage()Z (BPlusTree.java:4641)
  at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next()Z (BPlusTree.java:4570)
  at org.apache.ignite.internal.processors.query.h2.H2Cursor.next()Z (H2Cursor.java:78)
  at org.h2.index.IndexCursor.next()Z (IndexCursor.java:305)
  at org.h2.table.TableFilter.next()Z (TableFilter.java:499)
  at org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow()[Lorg/h2/value/Value; (Select.java:1452)
  at org.h2.result.LazyResult.hasNext()Z (LazyResult.java:79)
  at org.h2.result.LazyResult.next()Z (LazyResult.java:59)
  at org.h2.command.dml.Select.queryFlat(ILorg/h2/result/ResultTarget;J)Lorg/h2/result/LazyResult; (Select.java:519)
  at org.h2.command.dml.Select.queryWithoutCache(ILorg/h2/result/ResultTarget;)Lorg/h2/result/ResultInterface; (Select.java:625)
  at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(ILorg/h2/result/ResultTarget;)Lorg/h2/result/ResultInterface; (Query.java:114)
  at org.h2.command.dml.Query.query(ILorg/h2/result/ResultTarget;)Lorg/h2/result/ResultInterface; (Query.java:352)
  at org.h2.command.dml.Query.query(I)Lorg/h2/result/ResultInterface; (Query.java:333)
  at org.h2.command.CommandContainer.query(I)Lorg/h2/result/ResultInterface; (CommandContainer.java:113)
  at org.h2.command.Command.executeQuery(IZ)Lorg/h2/result/ResultInterface; (Command.java:201)
  at org.h2.jdbc.JdbcPreparedStatement.executeQuery()Ljava/sql/ResultSet; (JdbcPreparedStatement.java:111)
  at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(Ljava/sql/Connection;Ljava/sql/PreparedStatement;ILorg/apache/ignite/internal/processors/query/GridQueryCancel;)Ljava/sql/ResultSet; (IgniteH2Indexing.java:961)
  at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(Ljava/sql/PreparedStatement;Ljava/sql/Connection;Ljava/lang/String;Ljava/util/Collection;ILorg/apache/ignite/internal/processors/query/GridQueryCancel;)Ljava/sql/ResultSet; (IgniteH2Indexing.java:1027)
  at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(Ljava/sql/Connection;Ljava/lang/String;Ljava/util/Collection;ZILorg/apache/ignite/internal/processors/query/GridQueryCancel;)Ljava/sql/ResultSet; (IgniteH2Indexing.java:1006)

alamar alamar · Accepted Answer · 2019-04-11T11:42:53

Is there a chance that you have tried to execute SELECT * without WHERE clause or similar request with huge result set? Result set will be retained on heap which will lead to OOM when serving such request.

Either use LIMIT clause in your SQL query, or use lazy=true in your Connection/SqlFieldsQuery.

Getting out of memory on Ignite server node

1 Answers