There is a fundamental problem with the way node.js works and the way TokuMX multi-statement transactions work.
In TokuMX, multi-statement transactions (with beginTransaction) are associated with the client connection that runs the command. Any operations done on the same connection after that, until the next rollbackTransaction or commitTransaction command, count as part of the transaction.
If you start a transaction and some other thread uses your connection, its operations will become part of your transaction. Also, if you start a thread and then switch to a different connection, those operations after the switch won't be properly associated with a transaction.
The reason we chose this model was that it made sense for the majority of drivers. With most drivers in languages with explicit threading models, that use connection pooling, there is a driver mechanism to reserve a connection for a given thread of execution, e.g. start_request in PyMongo or RequestStart in the MongoDB C# driver. In these drivers, any thread that wants to use a TokuMX transaction must do so within the context of such a reserved connection.
In node.js, with the current drivers (AFAIK right now) there is no way for a logical group of code to associate itself exclusively with a single transaction. This is partly because there is not a good concept in the node.js execution model itself of tying a logical group of code together (through multiple callbacks) as a "thread" to which a connection could be associated. Briefly, there's no way to tell node "execute the beginTransaction command, make sure nobody else uses this connection, and when you call my callback, make sure I get a reference to the connection so I can use it for more things".
Now, there is still a set of transactional semantics you can get from TokuMX with the node.js driver if you exclude multi-statement transactions. All queries still use MVCC snapshots, and all batched inserts, and multi-document updates and deletes are serializably isolated from each other, and are atomic even across multiple documents. Keep in mind that these are relaxed in a sharded setting; transactional semantics don't cross shard boundaries at this time.
There is a project BlueRival/node-tokumx-native by someone who is reportedly working on this problem. According to one of their github issues they should be starting work on this problem soon, but I am not in direct contact with them so I can't speak to schedules or planning.
For now, I recommend you try using another driver or try to write your application in such a way that the single-statement transactional semantics of TokuMX are sufficient.
Disclaimer: I'm a TokuMX engineer.
