Unable to decrypt RSA-OAEP message encrypted with PyCryptodome using SubtleCrypto Web Crypto API

Question

On the server side I am using PyCryptodome to encrypt a message with RSA-OAEP (with SHA-256).
I'm trying to decrypt the message using SubtleCrypto Web Crypto API on the client side, but it give me a DOMException error without no further details.
On SubtleCrypto I can import the private key generated in PyCryptodome without problems, but it gives me the error when I'm trying to decrypt the message.

I have also tried to import the public key generated on PyCryptodome on client side to encrypt the same message with SubtleCrypto. In that case I can decrypt it without problems, using the same flow as before.

Are the RSA-OAEP algorithms between these two libraries incompatible? I noticed that PyCryptodome references RFC 8017(v2.2) and SubtleCrypto RFC 3447(v2.1) in their respective documentation.

Edit:

Server side code (pycryptodome==3.9.8):

  from Crypto.PublicKey import RSA
  from Crypto.Cipher import PKCS1_OAEP

  class Cipher:

  def rsa_encrypt(self, data, key_string):
      key = RSA.importKey(key_string)
      rsa_encryption_cipher = PKCS1_OAEP.new(key)
      ciphertext = rsa_encryption_cipher.encrypt(data)
      return base64.b64encode(ciphertext)

  def rsa_decrypt(self, data, key_string):
      data = base64.b64decode(data)
      key = RSA.importKey(key_string)
      rsa_decryption_cipher = PKCS1_OAEP.new(key)
      plaintext = rsa_decryption_cipher.decrypt(data)
      return plaintext

( ... )

Client side code

private decryptRSAString (encryptedText: string, privateKey: string) : Observable<ArrayBuffer> {

  return Observable.create ((observer: any) => {

    let keyBuffer: ArrayBuffer = this.str2ab(window.atob(privateKey));
    let encryptedTextBuffer: ArrayBuffer = this.str2ab(window.atob(encryptedText));
    let algorithmParams: RsaHashedImportParams = {
      name: "RSA-OAEP",
      hash: "SHA-256"
    };
    window.crypto.subtle.importKey(
      'pkcs8',
      keyBuffer,
      algorithmParams,
      true,
      ["decrypt"]
    ).then (
      (cryptoKey: CryptoKey) => {
        window.crypto.subtle.decrypt(
          {
            name: "RSA-OAEP"
          },
          cryptoKey,
          encryptedTextBuffer
        ).then (
          (decryptedMessage: ArrayBuffer) => {
            observer.next (decryptedMessage);
            observer.complete();
          },
          (error: any) => {
            observer.error (error)
          }
        )
      },
      (error: any) => {
        observer.error (error)
      }
    );
  });
}

Welcome to Stackoverflow. It could be helpfull to see the code you are struggling with, all other answer would be some kind of "arguing". — Michael Fehr
Edited with the client-side code. The code works decrypting messages encrypted with the same library, but it give a DOMException when I try to decrypt a message encrypted with PyCryptodome as I explained before. — Sergi Sánchez Centelles

Topaco Topaco · Accepted Answer · 2020-08-26T13:27:58

PyCryptodome does not apply SHA-256 as default digest for OAEP, but SHA-1, here. Accordingly SHA-1 must be used on the WebCrypto side:

let algorithmParams: RsaHashedImportParams = {
    name: "RSA-OAEP",
    hash: "SHA-1"
};

Of course you can also apply SHA-256 on the PyCryptodome side, then no changes are necessary on the WebCrypto side.

from Crypto.Hash import SHA256
...
rsa_encryption_cipher = PKCS1_OAEP.new(key, SHA256) # default: Crypto.Hash.SHA1

With consistent digests on both sides I can successfully decrypt a ciphertext with your WebCrypto code, which I have previously generated with your PyCryptodome code (using my own keys).

Unable to decrypt RSA-OAEP message encrypted with PyCryptodome using SubtleCrypto Web Crypto API

1 Answers