1
votes

I'm sending a base64 encoded string from a Classic ASP page to a JSP page. The string was RC4 encrypted, prior to being encoded.

Now, I observed that, within the ASP page, encoding and decoding the string with base64 works fine. However, the base64 decoded string on the JSP page is incorrect. I also tried decoding the string in Eclipse and got the same results. It seems to be related to the character encoding type, but I'm struggling with determining what is precisely the issue.

  • base64 encoded string: yOBIc4FY
  • base64 decoded string (from ASP page): ÈàHsX (correct)
  • base64 decoded string (from JSP page and Eclipse): ÈàHs?X (incorrect)

Java/JSP code:

import org.apache.commons.codec.binary.Base64;

String base64String = "yOBIc4FY";

byte[] decodedBase64Byte = Base64.decodeBase64(base64String);
        
// ÈàHs?X
decodedBase64String = new String(decodedBase64Byte, "ISO-8859-1");

// ÈàHs?X
decodedBase64String = new String(decodedBase64Byte, "windows-1252");

// ??Hs?X
decodedBase64String = new String(decodedBase64Byte, "utf-8");

To reiterate, the correct value should be ÈàHsX. I don't understand what is the problem. Any help would be appreciated.

Thank you.

Update

Let me expound on this further.

The RC4 crytographic algorithm in Classic ASP is widely available, so I won't waste real estate posting it here. However, I will show the base64 encoder/decoder I'm using for `Classic ASP below.

For RC4, the plaintext value I'm using is foobar. And the key I'm using is test. Ostensibly, decoding the base64 string should return the cipher. And decrypting the cipher should return the plaintext value.

' Functions to provide encoding/decoding of strings with Base64.
' 
' Encoding: myEncodedString = base64_encode( inputString )
' Decoding: myDecodedString = base64_decode( encodedInputString )
'
' Programmed by Markus Hartsmar for ShameDesigns in 2002. 
' Email me at: [email protected]
' Visit our website at: http://www.shamedesigns.com/
'

    Dim Base64Chars
    Base64Chars =   "ABCDEFGHIJKLMNOPQRSTUVWXYZ" & _
            "abcdefghijklmnopqrstuvwxyz" & _
            "0123456789" & _
            "+/"


    ' Functions for encoding string to Base64
    Public Function base64_encode( byVal strIn )
        Dim c1, c2, c3, w1, w2, w3, w4, n, strOut
        For n = 1 To Len( strIn ) Step 3
            c1 = Asc( Mid( strIn, n, 1 ) )
            c2 = Asc( Mid( strIn, n + 1, 1 ) + Chr(0) )
            c3 = Asc( Mid( strIn, n + 2, 1 ) + Chr(0) )
            w1 = Int( c1 / 4 ) : w2 = ( c1 And 3 ) * 16 + Int( c2 / 16 )
            If Len( strIn ) >= n + 1 Then 
                w3 = ( c2 And 15 ) * 4 + Int( c3 / 64 ) 
            Else 
                w3 = -1
            End If
            If Len( strIn ) >= n + 2 Then 
                w4 = c3 And 63 
            Else 
                w4 = -1
            End If
            strOut = strOut + mimeencode( w1 ) + mimeencode( w2 ) + _
                      mimeencode( w3 ) + mimeencode( w4 )
        Next
        base64_encode = strOut
    End Function

    Private Function mimeencode( byVal intIn )
        If intIn >= 0 Then 
            mimeencode = Mid( Base64Chars, intIn + 1, 1 ) 
        Else 
            mimeencode = ""
        End If
    End Function    


    ' Function to decode string from Base64
    Public Function base64_decode( byVal strIn )
        Dim w1, w2, w3, w4, n, strOut
        For n = 1 To Len( strIn ) Step 4
            w1 = mimedecode( Mid( strIn, n, 1 ) )
            w2 = mimedecode( Mid( strIn, n + 1, 1 ) )
            w3 = mimedecode( Mid( strIn, n + 2, 1 ) )
            w4 = mimedecode( Mid( strIn, n + 3, 1 ) )
            If w2 >= 0 Then _
                strOut = strOut + _
                    Chr( ( ( w1 * 4 + Int( w2 / 16 ) ) And 255 ) )
            If w3 >= 0 Then _
                strOut = strOut + _
                    Chr( ( ( w2 * 16 + Int( w3 / 4 ) ) And 255 ) )
            If w4 >= 0 Then _
                strOut = strOut + _
                    Chr( ( ( w3 * 64 + w4 ) And 255 ) )
        Next
        base64_decode = strOut
    End Function

    Private Function mimedecode( byVal strIn )
        If Len( strIn ) = 0 Then 
            mimedecode = -1 : Exit Function
        Else
            mimedecode = InStr( Base64Chars, strIn ) - 1
        End If
    End Function

From within ASP, the plaintext value is correctly realized from the cipher:

plainText: foobar

cipherText: ÈàHsX

base64 String: yOBIc4FY

Decoded base64 String: ÈàHsX

Decrypted text: foobar

However, passing the cipher as a base64 string to JSP/Java, the JSP/Java looks like this:

plainText: foobar (from ASP)

cipherText: ÈàHsX (from ASP)

base64 String: yOBIc4FY

Decoded base64 String: ÈàHs?X

Decrypted text: foobßr

So, something is not adding up right here. In fact, for Java, making one change in how I decrypt the decipher returns the proper decrypted text of foobar. The RC4 decryption in the Java code takes the cipher in the form of type int[].

public int[] decrypt(int[] ciphertext, byte[] key) throws Exception {
    return encrypt(ciphertext, key);
}

In other words, I have to convert the cipher from type String to type int[]. I use the function below to do just that:

public static int[] convertToIntArray(byte[] input)
{
    int[] ret = new int[input.length];
    for (int i = 0; i < input.length; i++)
    {
        ret[i] = input[i] & 0xff; // Range 0 to 255
    }
    return ret;
}

I have two choices. I can decode the base64 string as type byte[] and decrypt, which will return foobar.

String base64String = "yOBIc4FY";

byte[] decodedBase64Byte = Base64.decodeBase64(base64String);

int[] cipheredText =  convertToIntArray(decodedBase64Byte);

Or, I can decode the base64 string as type byte[] and then convert it to type String and back again to type byte[] to decrpyt, which will return foobßr.

String base64String = "yOBIc4FY";

byte[] decodedBase64Byte = Base64.decodeBase64(base64String);

// ÈàHs?X
String decodedBase64String = new String(decodedBase64Byte, "ISO-8859-1");

int[] cipheredText =  convertToIntArray(decodedBase64String.getBytes());

My guess is then the original byte sequence is correct, since the RC4 decryption function successfully returns foobar. However, when I convert the byte sequence to a String of some character encoding set, it changes the value, ultimately with a decrypted value of foobßr.

It still doesn't make sense then why ASP and JSP/Java are reporting slightly different cipher values? ASP has no trouble decoding the base64 string or the cipher back into its plaintext value. I can't tell if the issue is with ASP, JSP, or both.

1
Which web server are you using? Tomcat?rickz
Yeah, Tomcat for JSP. I also tested it locally in Eclipse (JDK/JRE 1.6-1.8) and same problem.user3621633

1 Answers

2
votes

The correct decoding of yOBIc4FY is 6 bytes, specifically:

c8 e0 48 73 81 58

The ÈàHsX value probably just ignores the character 0x81 as unprintable.

Proof:

y      O      B      I      c      4      F      Y
110010 001110 000001 001000 011100 111000 000101 011000

11001000 11100000 01001000 01110011 10000001 01011000
c8       e0       48       73       81       58

To address your follow-up question - you should use the byte array you get from the base64 decoder. Convert it to int[] if you need, but don't create a String out of it, because the encoding will mess it up:

static void printByteArray(byte[] bytes) {
    for (byte b : bytes) {
        System.out.print(Integer.toHexString(b & 0xff) + ", ");
    }
    System.out.println();
}

public static void main(String[] args) {

    byte[] cipherBytes = Base64.getDecoder().decode("yOBIc4FY");
    printByteArray(cipherBytes); // c8, e0, 48, 73, 81, 58 - correct

    cipherBytes = new String(cipherBytes).getBytes();
    printByteArray(cipherBytes); // c8, e0, 48, 73, 3f, 58 - wrong
    // your results may vary depending on your default charset,
    // these are for windows-1250
}

Here you can see that the original correct byte 0x81 was changed into a question mark ? (byte 0x3f), because 0x81 doesn't represent a valid character in the charset used when creating the String from the byte array.