Data Load to Ignite cache is very slow from Oracle Table

Question

I am trying to load data from Oracle Table to Ignite Cache using Cache Store Load Cache method.

Following is the logic implemented in the Load Cache method to load the data from Oracle Table using Ignite Cache.

JDBC connector is used to connect to Oracle Table and the data is available in Result Set Cursor.
While loop is used to loop through the Result Set Object and insert the data into the cache.

Is there any other way to insert the data from Oracle Table to Ignite Cache. If possible please share sample code.

Its taking almost 1hour to load the 0.1 million data ignite Cache using result set Cursor — Nithin Govindaraju
Do you have your own implementation of cache Store? Does switching to CacheJdbcPojoStore help? — alamar

Andrei Aleksandrov Andrei Aleksandrov · Accepted Answer · 2020-01-27T10:25:15

Here you can see one of possible variant that used Spark:

1)Load data from Oracle using Spark Data Frames

https://dzone.com/articles/read-data-from-oracle-database-with-apache-spark

2)Store the values from Oracle data frame to Ignite using Ignite Spark integration:

https://apacheignite-fs.readme.io/docs/ignite-data-frame

String configPath = "client.xml";

SparkConf sparkConf = new SparkConf()
 .setAppName("Example");

SparkSession session = SparkSession.builder()
 .config(sparkConf)
 .getOrCreate();

Dataset < Row > oracleDataset = ...; //Read from Oracle DB

DataFrameWriter < Row > df = oracleDataset
    .write()
    .format(IgniteDataFrameSettings.FORMAT_IGNITE())
    .option(IgniteDataFrameSettings.OPTION_CONFIG_FILE(), configPath)
    .option(IgniteDataFrameSettings.OPTION_TABLE(), "Person")
    .option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS(), "id, city_id")
    .option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PARAMETERS(), "template=partitioned,backups=1")
    .mode(Append);

df.save();

session.close();

In the same way, you can use another streaming tools e.g Ignite Kafka streamer.

https://rmoff.net/2018/12/12/streaming-data-from-oracle-into-kafka-december-2018/

https://apacheignite-mix.readme.io/docs/kafka-streamer

All Ignite integrations you can see here:

https://apacheignite-mix.readme.io/docs/getting-started

Data Load to Ignite cache is very slow from Oracle Table

1 Answers