Ignite consumes all memory and fails with OutOfMemory when iterating over cache

2
votes

I'm trying to iterate over all cache entities using ScanQuery and iterator (not to pull them from distributed cache to local client at once):

IgniteCache cache = ignite.getOrCreateCache("test2");
ScanQuery<Integer, Person> scan = new ScanQuery<>();
scan.setPageSize(256);
Iterator<Cache.Entry<Integer, Person>> it = cache.query(scan).iterator();
int id;
while(it.hasNext()) {
    id = it.next().getValue().getId();
    <...>
}

but the code above fails, consuming all memory available. In the same time it works well when I'm trying to get iterator from cache:

    IgniteCache cache = ignite.getOrCreateCache("test2");
    Iterator<Cache.Entry<Integer, Person>> it = cache.iterator();
    int id;
    while(it.hasNext()) {
        id = it.next().getValue().getId();
        <...>
    }

Docs state that:

QueryCursor represents query result set and allows for transparent page-by-page iteration. Whenever user starts iterating over the last page, it will automatically request the next page in the background.

So why ignite local node fails when iterating over cache with ScanQuery?

UPD:

  1. Person is an example name instead of actual class's name. Actual class contain one Integer and 10 String fields.
  2. Actually I've already set page size to a smaller number - 256 instead of default 1024. Same behavior with default and smaller value
  3. When I try to use cache.query(scan).getAll() things go same way, but I can't use iterator value in while loop, application just became stuck until OOM.

  4. Exception msg:

    Aug 31, 2018 6:16:15 PM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=105, name=tcp-disco-sock-reader-#13, nodeId=83e986dc-9fc1-433c- 8953-2a11460376a0] java.lang.OutOfMemoryError: Java heap space

full stacktrace: https://pastebin.com/raw/ZpHnRjx8

2
javaignite
Several questions: What does Person look like, how big is that class? What happens if you don't set pageSize (or set a different value)? (Do you know what pageSize does?) What happens if you do cache.query(scan).getAll()? And last but not least, please add the full exception message and stacktrace. – Max Vollmer
@Max I've updated post. – aryndin
Possible duplicate of Error java.lang.OutOfMemoryError: GC overhead limit exceeded – Max Vollmer
Seems like this is caused by the garbage collector spending too much time trying to find and free all the small temporary objects that get created by your code. See the other question for possible solutions. – Max Vollmer
@Max I think this question is more ignite related than java related. By some reasons everything works fine when I get iterator from cache and don't use ScanQuery – aryndin

2 Answers

1
votes

It looks like a known issue https://issues.apache.org/jira/browse/IGNITE-8892 It is already fixed and will be available in Apache Ignite 2.7 Can you please check your code with the latest master branch?

0
votes

I have also experienced this behaviour with ScanQuery; there appears to be a memory leak in the cursor where it holds on to references to iterated cache objects. As a workaround you may be able to use the SqlQuery interface which doesn't exhibit the same behaviour, assuming your caches are setup for SQL access:

SqlQuery<Integer, Person> scan = new SqlQuery<>(Person.class, "1=1");