Apache Ignite performance

0
votes

I am running a POC of Ignite vs. stored procedure performance in doing some computation in a batch job. The batch job has the following steps:

  1. Read from the cache (SELECT with INNER JOIN (2 tables on primary key) and a simple WHERE clause (a date column on the smaller table))
  2. In a loop, do some calculation (multiplication) and write result back to the cache.

The # of records returned in #1 is proportional to the # of records written in #2. The RDBMS compared with is SQL Server.

I have implemented the code that uses Ignite for the read and write operations. I implemented the same operations in a stored procedure. When I ran the test I found that Ignite is so much slower: on a size of 220,000 records here is the total time to run the entire job:

  • SQL Server stored procedure: 9 seconds
  • Ignite: 2 hours 40 minutes

Notes:

  • Ignite configuration is set to CacheAtomicityMode.TRANSACTIONAL and CacheMode.LOCAL, on a single server node. The transaction boundary is before and after each step.
  • I verified that results are the same, ie. they are doing the same thing, there is no coding issue
  • The machine has 16GB RAM, CPU is an i7

Before running the test I was already expecting Ignite to be slower due to overhead, and nothing else can really beat a stored procedure running in the database (eg. due to db optimizations). However I am surprised that the difference is this big. Also, for the entire Ignite job my CPU fan was whirring nonstop, which is weird because the computation is a simple multiplication and the cache is in RAM (which is silent).

I also ran the job in CacheMode.PARTITIONED mode (2 server nodes, one of them remote, + 1 client), on a data set is slightly smaller (145k records). The total run time for this is 1 hr 8 minutes. Projecting this time to 220k records (same as initial set) it is 1 hour 43 seconds (approx), which is faster than CacheMode.LOCAL, but still, significantly slower than expected for this kind of volume. (220k is not a lot)

It would be great if you can tell me any configuration I need to implement or check? Thank you!

1
performanceignitein-memory-database

1 Answers

0
votes

In Ignite, stored procedures come in the flavour of Compute Grid tasks. Can you try to rewrite your test case in terms of tasks?

If you are using transactions, keep in mind SQL currently non-transactional, so you should probably use transactional put/get.

I also think you should definitely use PARTITIONED, since it means you can add nodes while keeping reliability, which single-server SQL cannot provide. Keep in mind it is important to collocate data if you are doing joins, that means aligning data by affinity keys.