Error encountered in the "session.sendAndReceiveWithRetry(tx)"

Question

our app is using Corda V2.0 currently, and is deployed to dedicated server, we are using 3 notaries with RAFT configured, but when there is an input state of the transaction, below error was found in the log

messaging.NodeMessagingClient.sendWithRetry - Reached the maximum number of retries (3) for message ClientMessageImpl

And after reading the code of FinalityFlow, we found this error should happened in the step "session.sendAndReceiveWithRetry(tx)" of below codes in NotaryFlow.kt

val response = try {
                val session = initiateFlow(notaryParty)
                if (serviceHub.networkMapCache.isValidatingNotary(notaryParty)) {
                    subFlow(SendTransactionWithRetry(session, stx))
                    session.receive<List<TransactionSignature>>()
                } else {
                    val tx: Any = if (stx.isNotaryChangeTransaction()) {
                        stx.notaryChangeTx
                    } else {
                        stx.buildFilteredTransaction(Predicate { it is StateRef || it is TimeWindow || it == notaryParty })
                    }
                    session.sendAndReceiveWithRetry(tx)
                }
            } catch (e: NotaryException) {
                if (e.error is NotaryError.Conflict) {
                    e.error.conflict.verified()
                }
                throw e
            }

We tried to ping each other between the node, notary 1, 2 and 3 with RPC port, all of them are listening, but from the error message, we are not sure whether this is caused by the communication/messaging could not work between node and notaries. Notaries node are running and there is no any error in their logs.

So we would like to know whether ping each other with the RPC port can ensure the messaging can work well between them? Is there any approach can test whether messaging between two nodes(node and notary) can be sent and received properly? Thanks a lot.

This is a bug. It is being tracked here: r3-cev.atlassian.net/browse/CORDA-1527. — Joel

Joel Joel · Accepted Answer · 2018-06-05T13:27:50

There is a known bug in Corda 3 whereby after rebooting a Raft notary cluster, it no longer services client requests. Instead, the Raft cluster ends up partitioned with multiple leaders.

This issue is being tracked in Corda's JIRA board here: https://r3-cev.atlassian.net/browse/CORDA-1527.

Error encountered in the "session.sendAndReceiveWithRetry(tx)"

1 Answers