C# AES Decryption returns 0x3F (0011 1111) when byte value > 127

Question

I'm trying to encrypt/decrypt a binary file with AES 256. When I encrypt it there's no problem with the bytes, apparently. But, when I decrypt the encrypted binary file it returns a new binary file with a lot of bytes = 0x3F (0011 1111). I think these bytes are > 127 value, because the other bytes thats decrypts correctly are <= 127 value.

I think there's a sign issue, but I'm using the byte type, that accepts 0 to 255 value. Other possibility is a encoding issue.

My Decrypt function:

                static public byte[] AESDecrypt(byte[] ct, byte[] key, byte[] IV)
    {
        byte[] ret = new byte[ct.Length];

        Aes AES = Aes.Create();
        ICryptoTransform Decryptor = AES.CreateDecryptor(key, IV);

        using (MemoryStream ms = new MemoryStream())
        {
            using (CryptoStream cs = new CryptoStream(ms, Decryptor, CryptoStreamMode.Write))
                cs.Write(ct, 0, ct.Length);

            ret = ms.ToArray();
        }

        return ret;
    }

The cs.Write() already returns an int value = 0x3F for each byte > 127.

The function call:

byte[] pt = { 128, (byte)'t', (byte)'e', (byte)'s', (byte)'t' };
        byte[] ct = AESEncrypt(pt, AESStorageKey, AESStorageIV);
        byte[] pt2 = AESDecrypt(ct, AESStorageKey, AESStorageIV);
        txtPassword.Text = pt[0].ToString() + " " + ct[0].ToString() + " " + pt2[0].ToString();

The output in the TextBox is: "128 183 63" Note that 63 is the 0x3F value (0011 1111), but the correctly return should be 128 instead.

Ok, with your help about StreamReader/StreamWriter and binary handling I found the error. That was at the Encrypt function, because I was using StreamWriter to handle binary data. Now I have fixed it! Thank you all!

The Encrypt Function:

        static public byte[] AESEncrypt(byte[] pt, byte[] key, byte[] IV)
    {
        byte[] ret;

        AesManaged AES = new AesManaged();
        ICryptoTransform Encryptor = AES.CreateEncryptor(key, IV);

        using (MemoryStream ms = new MemoryStream())
        {
            using (CryptoStream cs = new CryptoStream(ms, Encryptor, CryptoStreamMode.Write))
                cs.Write(pt, 0, pt.Length); //The correct binary handling

            // Here was the error before:
            //using (StreamWriter sw = new StreamWriter(cs))
            //  sw.Write(Encoding.ASCII.GetString(pt));

            ret = ms.ToArray();
        }

        return ret;
    }

If you're passing bytes in to this method, and retrieving bytes out of this method, why are you messing about with strings at all? Remove any stringly-typed code from both your encode and decode paths and be far happier. — Damien_The_Unbeliever
I'm trying to encrypt/decrypt a binary file... then StreamReader should not be used and will only corrupt your data. — President James K. Polk

Thinh Tran Thinh Tran · Accepted Answer · 2020-07-07T03:05:15

StreamReader by default using UTF-8. I think you should use Encoding.UTF8.GetBytes() instead.

C# AES Decryption returns 0x3F (0011 1111) when byte value > 127

1 Answers