I am using Apache Ignite 2.6. I am using Ignite Filesystem, and when I write a specific file, which is about 25 MB, to IGFS, over and over, the data is not saved into the non-heap space. Instead, it goes into heap, which is subject to Garbage Collection, and it is relatively slow. How do I get IGFS to save a file into the large heap space I have allocated for it?
High level architecture--I have a client ignite node running inside of a tomcat for now, and a server ignite node, on which I intend this data to be stored. Scaling can occur once I get this working as expected--but it is very slow because of the aforementioned problem. It also OOMs when it runs out of heap space very quickly. Thing is, I want it to use the 30G of NON HEAP space I have allocated!
I intend this it to be an in memory cache. I am allocating 2 G of heap space and 30G of non heap space to the JVM. The non heap space never gets used and it runs out of memory as a result. I have confirmed that the non-heap space is not used using the JMX Console Memory tab--non heap space stays well below 100M, while heap space quickly balloons to 2G and then the JVM crashes.
The details: First, my ignite configuration (spring xml):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_FALLBACK"/>
<property name="searchSystemEnvironment" value="true"/>
</bean>
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="marshaller">
<bean class="org.apache.ignite.internal.binary.BinaryMarshaller" />
</property>
<property name="fileSystemConfiguration">
<list>
<bean class="org.apache.ignite.configuration.FileSystemConfiguration">
<property name="name" value="igfs"/>
<property name="blockSize" value="#{128 * 1024}"/>
<property name="perNodeBatchSize" value="512"/>
<property name="perNodeParallelBatchCount" value="16"/>
<property name="prefetchBlocks" value="32"/>
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration" >
<!-- if I don't set this, the system region runs out of memory almost immediately -->
<property name="systemRegionMaxSize" value="#{6L * 1024 * 1024 * 1024"} />
<property name="systemRegionInitialSize" value="#{6L * 1024 * 1024 * 1024"} />
</bean>
</property>
</bean>
Here is the script I use to start up my ignite server process. It's a shell script running on a Linux machine with 64 G RAM and 40 G disk space.
IGNITE_HOME=/data/apache-ignite
export IGNITE_HOME
IGNITE_JMX_PORT=1234
export IGNITE_JMX_PORT
$IGNITE_HOME/bin/ignite.sh $IGNITE_HOME/ignite-media-server.xml -J-Xmx2G -J-Xms2G -J-XX::+HeapDumpOnOutOfMemoryError -J-XX:HeapDumpPath=$IGNITE_HOME -J-XX:+PrintGC -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps -J-Xloggc:$IGNITE_HOME/gc.log-$(date +%m%d-%H%M%S) -J-XX:+UseG1GC -J-XX:DisableExplicitGC -J-XX:MaxDirectMemorySize=30G
This is the code that creates my client igfs object, through which I save files to ignite. They tend to be on the large side.
public void init() throws Exception{
igniteInstanceName = "client-name=" + hostInfo.getLocalHost();
Ignition.setClientMode(true);
// reading in the same config file as the server uses to start up above. The big difference is the clientMode set to true here.
try(InputStream configFileInputStream = new FileInputStream(ResourceUtils.getFile("ignite-media-server.xml"));){
ignite = IgnitionEx.start(configFileInputStream, igniteInstanceName, null, null);
igfs = ignite.fileSystem("igfs");
}
catch(Throwable t){ /* do log */}
}
Here is a save method, that saves my files to ignite:
public saveStream(String cachePath, AudioInputStream toCache){
OutputStream os = null;
try{
IgfsPath cacheFile = new IgfsPath(cachePath);
os = igfs.create(cacheFile, true);
AudioSystem.write(toCache.getDataStream, AudioFileFormat.TYPE.WAVE, os);
}
finally{
// close streams
}
}
Why doesn't my data get saved to the speedy off-heap space? What am I missing? my server.config comes almost straight from the igfs provided example.
In other confusion, when I use ignitevisor.cmd to inspect memory usage on the server node before and after a shorter test (that doesn't make it crash) I see the following:
Look at memory allocation while ignite is empty in ignitevisor.cmd. See that my igfs region says:
- Heap Memory Initialized: 2g
- Heap Memory Used: 56mb
- Non-heap memory initialized: 2mb
- Non Heap memory used: 49 mb
- Non heap memory maximum: 744mb
Create JUST SHY 2 G worth of files saved in IGFS--just short of an OOM since from bitter experience I know it will blow up shortly. Use ignitevisor.cmd to look at the memory allocation of the nodes. This is what ... – MeowCode 2 mins ago
- Heap memory initialized: 2gb
- Heap memory used: 1gb
- Non Heap memory used 64 MB
- Non heap memory maximum: 744mb
Why is there still almost nothing in non-heap? And why does ignitevisor think that the non-heap maximum is 744 MB when it should be 30 GB?
In other points of interest, if I increase my heap size to 6 GB, it runs longer, but still the server crashes with an "OutOfMemoryError:Java heap space". Interestingly, I can reproduce this even when I enable disk persistence. Inspecting the heap dump file reveals a lot of ConcurrentLinkedHashMap entries. The entries themselves are "org.apache.ignite.internal.GridTopic" objects. Each one has a uuid and most appear to be of type TOPIC_DATASTREAM.
