datomic and table join operation

Question

This question is regarding Datomic in the context of spark streaming. I have two tables T1 and T2. Updates happening on these 2 tables (updated record) may go to (get processed) any of two machines M1 and M2.

I want to take join of T1 and T2 records on M1 and M2 as and when new records of T1 or T2 arrive on M1 or M2. Here the problem is M1's and M2's cache may contain, some data of both T1 and T2.

If new T1 record arrives on M1 then on M1, T1 record will be joined with corresponding record of T2 (cached on M1, which could be stale). At the same this/corresponding T2 record got updated and received on M2. Now on M2, T2 record will be joined with corresponding record of T1 (cached on M2, could be stale) and which got just updated and being processed on M1.

In this situation both M1 and M2 will compute join of T1 and T2 records in incorrect way, because both use cached/stale values of records and changes in T1 and T2 record (novelty info propagation) though committed to backend DB through transactor, has not reached M1's and M2's cache.

Is this situation possible in Datomic and how to solve this? Also is Datomic a good choice for such types of use-cases?

Answer 1

This situation is not possible in Datomic, simply because Datomic by design runs the writer (transactor) on a single server/instance only, in a single process, in a single thread. Conversly, the readers (peers) are completely elastic.

I'm thinking Datomic might not be a good fit for your scenario since you are describing a multi-writer setup, something Datomic does not support. Datomic is operationally speaking generally a great fit if you want the capabilities Datomic provides and you don't need to do more than hundreds of writes per second to a single database.

The closest you get to high availability with Datomic, is a failover transactor, ready to take over if the main transactor fails.

http://docs.datomic.com/architecture.html

http://docs.datomic.com/ha.html

Answer 2

A way to solve this with Datomic might be to have the updates from T1 and T2 go into a Datomic database. Or you could have T1 go into T1 database, T2 into T2 database, optionally with a transactor for each.

Then M1 and M2 are peers of the database/s listening to the tx-queue or queues. The readers of the queues can claim the update for processing, for example by adding an attribute to the transaction it is processing, if it is not already there (which would indicate it is already claimed).

Whether this is a good choice or not is not something I can really answer with the provided info, but I put it here for comparisons with other solutions.