Different encryption Android vs pure Java - RSA/ECB/OAEPWithMD5AndMGF1Padding

Question

I encrypt a string in Android by the public key. However, I get an exception "Decryption error" when I try to decrypt the encrypted string by the private key in pure Java code. Can anyone help to find the problem?

Android code to encrypt

import android.util.Base64;
public static String encryptMessage(final String plainText, final PublicKey publicKey) throws Exception {
    Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithAndMGF1Padding");
    cipher.init(Cipher.ENCRYPT_MODE, publicKey);
    return Base64.encodeToString(cipher.doFinal(plainText.getBytes()), Base64.NO_WRAP);
}

Pure Java code to decrypt

import java.util.Base64;
public static String decryptMessage(final String encryptedText, final PrivateKey privateKey) throws Exception {
    Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithAndMGF1Padding");
    cipher.init(Cipher.DECRYPT_MODE, privateKey);

    Base64.Decoder decoder = Base64.getDecoder();
    byte[] byteArray = decoder.decode(encryptedText);
    byte[] decryptedArray = cipher.doFinal(byteArray);  // throw exception here
    String plainText = new String(decryptedArray);
    return plainText;
}

You may notice I have to use different Base64 APIs in Android and pure Java. I tried "RSA/ECB/PKCS1Padding", and it can decrypt correctly without the exception. Tried "RSA/ECB/OAEPWithSHA-256AndMGF1Padding" too but got the same exception.

Interesting. First of all, padding errors are also thrown if you use the wrong ciphertext / private key combination as modular exponentiation can always be performed. Can you please indicate the runtime environments you are using (Java version & Android API level)? — Maarten Bodewes
It's possible that both codes use different providers that apply different digests for OAEPWithMD5AndMGF1Padding. Here it must be taken into account that OAEP uses digests in two places: as the basis for the mask generation function MGF1 and for the hashing of the OAEP label (for details see RFC 8017). For decryption both digests must be identical to those for encryption. To test this, both digests can be explicitly defined with OAEPParameterSpec. — Topaco
pure Java built from IntelliJ IDEA 2019.1.4, Java SDK /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home. Android code built from Android Studio 3.6.3; compileSdkVersion 28, buildToolsVersion '28.0.3', minSdkVersion 21, targetSdkVersion 28, androidTestImplementation 'androidx.test.ext:junit:1.1.1', androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0', run AndroidTest in Samsung Android version 8.0.0. — user1443721
On my machine the BC Provider is used for Android, API level 28, OAEPWithMD5AndMGF1Padding, which applies MD5 for label and MGF. For Java 8, OAEPWithMD5AndMGF1Padding the SunJCE provider is used by default, which applies MD5 for the label but SHA1 for the MGF. Since the digests differ, the ciphertext generated with Android can't be decrypted with Java if OAEPWithMD5AndMGF1Padding is used in both codes. This problem can be eliminated with OAEPParameterSpec. It might be worthwhile to check the providers and, if necessary, adjust the digests with OAEPParameterSpec. — Topaco
@Topaco In Android on Samsung, cipher.getParameters().getParameterSpec(OAEPParameterSpec.class) throw exception "No provider found for RSA/ECB/OAEPWithAndMGF1Padding". In Java on my MAC, it works, and get MD5 for getDigestAlgorithm() and MGF1 for getMGFParameters(). — user1443721

Topaco Topaco · Accepted Answer · 2020-05-31T10:51:15

OAEP uses two digests, one for the OAEP label and a second as the basis for MGF1, see RFC 8017, 7.1. RSAES-OAEP, B.1. Hash Functions and B.2. Mask Generation Functions.

The issue is caused because the providers used on both sides of the OP code (Android / API Level 28 and Java 8) apply different MGF1 digests for OAEPWithMD5AndMGF1Padding.

On both sides the relevant parameters (provider, OAEP digest, MGF, MGF1 digest) can be determined after the initialization of the cipher e.g. with:

OAEPParameterSpec parameterSpec = cipher.getParameters().getParameterSpec(OAEPParameterSpec.class);
System.out.println("Provider: " + cipher.getProvider().getName());          
System.out.println("OAEP digest: " + parameterSpec.getDigestAlgorithm());  
System.out.println("OAEP MGF : " + parameterSpec.getMGFAlgorithm());        
System.out.println("OAEP MGF1 digest: " + ((MGF1ParameterSpec)parameterSpec.getMGFParameters()).getDigestAlgorithm());

With this, MD5 is determined as MGF1 digest on the Android side and SHA-1 on the Java side. On both sides MD5 is used as OAEP digest. The issue can be fixed if the digests are explicitly set with OAEPParameterSpec so that the same digests are used on both sides.

For example, the following code on the Java side ensures that MD5 is used as OAEP and MGF1 digest, analogous to the Android side.

OAEPParameterSpec oaepParameterSpecDec = new OAEPParameterSpec("MD5", "MGF1", new MGF1ParameterSpec("MD5"), PSource.PSpecified.DEFAULT);
cipher.init(Cipher.DECRYPT_MODE, privateKey, oaepParameterSpecDec);

The same applies to OAEPWithSHA-256AndMGF1Padding.

Note also that RFC 8017 in B.1. Hash Functions recommends SHA-1 and SHA-2 for RSAES-OAEP, but not MD5.

Different encryption Android vs pure Java - RSA/ECB/OAEPWithMD5AndMGF1Padding

1 Answers