When using wss endpoint, we were able to observe some errors and we were able to get rid of the errors with the following approach.
Please include the following parameter in the SecureWebSocketInboundEndpoint.xml file which resides in the <APIM_HOME>/repository/deployment/server/synapse-configs/default/inbound-endpoints directory.
TLSv1.1,TLSv1.2
Also, please remove the following parameters from the same SecureWebSocketInboundEndpoint.xml file if the following parameters(wss.ssl.trust.store.file and wss.ssl.trust.store.pass) exist in the file.
repository/resources/security/client-truststore.jks
wso2carbon
Please use the following sample web socket client to try out and run the WSS client. Please change the variable carbonKeyStoreLocation to point to <API-M_HOME>/repository/resources/security/wso2carbon.jks. Note that port for the WSS API is 8099.
You can download the WSS client in the following WSO2 official documentation under the WSS Support section. (In the second step.)[1] Further please change the access token, web socket endpoint and the carbonKeyStoreLocation with your one to try out the scenario.
[1] https://docs.wso2.com/display/AM260/Create+a+WebSocket+API