Roundtrip Unicode conversion returns different Byte[] array

Question

I'm tinkering with RSA signing of data.

I'm using a plaintext string, which i convert to byte array. i then generate private certificate, sign the byte array and then generate public key.

next i'm using the same byte array to verify the signature.

but i want to convert signature, in between steps, to the string - idea is to append it later on to the file that's being signed.

static void TestSigning(string privateKey)
    {
        string data = "TEST_TEST-TEST+test+TEst";
        Console.WriteLine("==MESSAGE==");
        Console.WriteLine(data);
        byte[] dataByte = Encoding.Unicode.GetBytes(data);
        using (var rsa = new RSACryptoServiceProvider())
        {
            rsa.FromXmlString(privateKey);
            var publicKey = rsa.ToXmlString(false);
            byte[] signature = rsa.SignData(dataByte, CryptoConfig.MapNameToOID("SHA512"));
            string signatureString = Encoding.Unicode.GetString(signature);
            byte[] roundtripSignature = Encoding.Unicode.GetBytes(signatureString);
            Console.WriteLine("==TEST==");
            Console.WriteLine(signature.Length.ToString());
            Console.WriteLine(roundtripSignature.Length.ToString());
            using (var checkRSA = new RSACryptoServiceProvider())
            {
                checkRSA.FromXmlString(publicKey);
                bool verification = checkRSA.VerifyData(
                    dataByte, 
                    CryptoConfig.MapNameToOID("SHA512"),
                    roundtripSignature);
                Console.WriteLine("==Verification==");
                Console.WriteLine(verification.ToString());
                Console.ReadKey();
            }
        }
    }

now here's the fun part if i use UTF8 encoding i get byte arrays of different length

256 is the original size

484 is the roundtrip

UTF7 returns different sizes too 256 vs 679

both ASCII and Unicode return proper sizes 256 vs 256.

i've tried using

var sb = new StringBuilder();
        for (int i = 0; i < signature.Length; i++)
        {
            sb.Append(signature[i].ToString("x2"));
        }

to get the string. I'm then using Encoding.UTF8.GetBytes() method

this time i get the sizes of: 256 vs 512 if i remove the format from toString() i get: 256 vs 670

signature verification alwayas failed. it works fine if i use 'signature' instead of roundtripSignature.

my question: Why, despite using same encoding type i get different byte arrays and strings? shouldn't this conversion be lossless?

The encryption is doing packing so repeating bits in data are compresses. So using UTF7 where MSB is set to zero will change filesize. Yes, no matter what encoding you are doing the un-encrypted data size should be the same at beginning and end of process. You should be able to open the xml text file and see what the differences are to help isolate issue. — jdweng
Signature bytes are not the encoding of any character set, so your method is not reliable. The standard way to convert crypto output to a string is to base64 encode it. — President James K. Polk
@JamesKPolk Thanks! that was it. Such simple solution and yet i was unable to find it. somehow I've assumed that those sets have a character for each combination of bits. — Xelbair

bartonjs bartonjs · Accepted Answer · 2017-06-08T15:11:26

Unicode isn't a good choice because, at minimum, \0, CR, LF, <delete>, <backspace> (and the rest of the control codes) can mess things up. (See an answer about this for Encrypt/Decrypt for more).

As @JamesKPolk said, you need to use a suitable binary-to-text encoding. Base64 and hex/Base16 are the most common, but there are plenty of other viable choices.

Roundtrip Unicode conversion returns different Byte[] array

1 Answers