Apache Ignite - issue in bulk load data from CSV files

Question

I am evaluating Apache Ignite for a specific use case. I am trying to load 50 GB of data to a Cache within Ignite. I have the data in CSV format. Currently I am reading data within a Java program in the form of Java Beans and adding beans to the cache using streamer API in Ignite. I am looking for alternative approches/options.

1. Do Ignite has any out-of-box feature available for bulk load the data in CSV files in a parallelized way (both read CSV and write to Cache)?
2. While loading the data using streamer API in a client program in Java, I am seeing the streamer object getting closed once around 600MB data added to it and causing the client node crash and successively other nodes crashing. The data load worked fine while processing 500MB of data. I am using a 3 physical node cluster and 2 ignite node instances in each of the servers.

Any suggestion will help.

Answer 1

1. There is no out-of-the-box CSV streamer.
2. Looks like after 500MB you are running out of memory. Can you check your JVM memory utilization with VisualVM and add bigger -Xmx value if you are running low?

Answer 2

Apache Ignite is in-memory cache. Do you have 50Gb+ memory?

I would like to propose another solution. Load your data from CSV to any database, for example H2 (add index to key field for performance). And configure Ignite cache to use cache store linked to database + readThrough option. You can load some data (not ALL 50Gb) via cache.load().

When your app will not find key in cache it will be transparently fetched from underlying store and cached.

What do you think about this approach?