The bit about "my keystore on the client side has only the application personal cert" is troubling. That won't work. The client KDB must have the server's public key. If the MQ server has SSLCAUTH(OPTIONAL) the the server's public cert is all that's required in the KDB for the connection to succeed.
The first part of the TLS handshake is where the client validates the server's certificate. The use of public/private key pairs is how the authenticity of the thing on the other side is assured. In order for that to happen the server must have its own personal certificate and the client must have the public key of the root of the signer chain. In the case of self-signed certificates the public portion of the personal cert must be trusted by the client. In the case of CA-signed certs the CA Root must be trusted by the client. Whichever one it is, the client must trust something used to sign the server's personal cert or that cert can't be validated.
The TLS handshake is symmetrical so the second part works exactly the same as the first but the roles are reversed. Therefore, in cases where mutual authentication is enabled the client must have its own personal cert (because that contains the private key) and the server must trust whatever signed the client's matching public key. If the client cert is self-signed the QMgr has to trust it. If the client's cert is CA-signed, the QMgr must trust the signer. Either way, the QMgr has to have a cert to validate the client in its KDB.
Following this logic, for anonymous client connections the required parts are the personal certificate in the QMgr's keystore (because it contains the QMgr's private key), and the matching trusted certificate in the client's KDB or for Java the Trust Store. Neither of these is optional.
If the client is to be authenticated you still need the same two certs as for anonymous clients because that part of the handshake has to complete before the client is authenticated. In addition, now you also need the client to have its own personal certificate (because it contains the client's private key) and the QMgr now needs to trust whatever signed the client's certificate - the client cert if self-signed or the signer root if CA-signed.
As a side note, There is also some confusion in the post because it says that "My WMQ 7.5 client application is C code and the WMQ server is running Java." There is nothing in the queue manager that uses Java on the server side. There are Java components installed to do things like manage JNDI objects and run the sample code. In modern MQ versions Java runs the Web Console. But there are no Java components in the QMgr itself and no Java components in the path of the incoming channel connection request. That's all habdled by the QMgr's listener, agent, and other internal processes. So I'm not at all sure what is being referred to there other than that the notion of Java running on the MQ server side and participating in the TLS handshake might be a source of some of the confusion mentioned in the post. ;-)
