c# -> Kafka over TLS: “The message received was unexpected or badly formatted”

Question

I'm trying to connect to Kafka having TLS enabled from C# client and getting exception "The message received was unexpected or badly formatted" during call to sslStream.AuthenticateAsClient(). Unfortunately none of posts in internet helped me so far to resolve the issue. Any idea what can be wrong?

Here is minimal sample C# code I used to initiate connection

namespace test_tls {
    class Program {
        static string clientCertificateFile = "C:\\Temp\\<CLIENT_CERTIFICATE_FILE>.crt";
        static X509Certificate2 clientCertificate = new X509Certificate2(clientCertificateFile);

        static void Main(string[] args) {
            var clientCertificateCollection = new X509Certificate2Collection(new X509Certificate2[] { clientCertificate });

            try {
                using( var client = new TcpClient("<IP_ADDRESS>", 9093) )
                using( var sslStream = new SslStream(client.GetStream(), false, CertificateValidator) ) {

                    sslStream.AuthenticateAsClient("<TARGET_HOST_NAME_AS_IN_THE_CERTIFICATE>",
                        clientCertificateCollection, SslProtocols.Tls, false);

                    //send/receive from the sslStream
                }
            }
            catch( Exception e ) {
                Console.Out.WriteLine(e);
                Console.Out.WriteLine("\n\n\nPress ENTER to exit");
                Console.In.ReadLine();
            }
        }

        static bool CertificateValidator(Object sender, 
                X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) {
            if( sslPolicyErrors == SslPolicyErrors.None ) {
                return true;
            }
            if( sslPolicyErrors == SslPolicyErrors.RemoteCertificateChainErrors ) {
                //we don't have a proper certificate tree
                return true;
            } 
            return false;
        }
    }
}

You can also get this if you have a protocol mismatch, like if the server expects TLS 1.3 but the client doesn't support it. — ESG

Xtra Coder Xtra Coder · Accepted Answer · 2016-08-09T14:30:34

And after some digging it appears that error message is absolutelly misleading and the root cause of the problem is - 'private key of client certificate' was missing during connection process.

X509Certificate2 should either be loaded using this way

string clientCertificateFile = "C:\\path\\to\\my.certificate.pfx";
X509Certificate2 clientCertificate = new X509Certificate2(clientCertificateFile, "<password>");

or from local certificate store (and it should be imported there with private key)

X509Store store = new X509Store(StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
X509Certificate2 cert = store.Certificates.Find(X509FindType.FindBySubjectName, "<Certificate 'Issued To' name>", false)[0];

Note: if you have you certificate and private key in separate files, this command can be used to merge them into PFX file

openssl pkcs12 -export -in my.cer -inkey my.key -out mycert.pfx

c# -> Kafka over TLS: “The message received was unexpected or badly formatted”

1 Answers