phpseclib RSA Decryption with openssl

Question

I used phpseclib for encryption, have some problem with openssl decrypt

//generate keys
extract($rsa->createKey(1024));
file_put_contents("public.pem",$publickey);
file_put_contents("private.pem",$privatekey);

encrypt text

    $rsa->loadKey($public_key); // public key
    $plaintext = '...';
    $ciphertext = $rsa->encrypt($plaintext);
    file_put_contents("ciphertext.txt",$ciphertext);

So in linux i'm decrypting :

xxd -p ciphertext.txt | tr -d '\n' 

And finally like the following command

openssl rsautl -decrypt -inkey private.pem

get this error :

error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190:
error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:674:

I am doing these settings for phpseclib :

define('CRYPT_RSA_PKCS15_COMPAT', true);

and now get this error on openssl:

error:04065084:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data too large for modulus:rsa_eay.c:590:

public key :

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGCglgIcCG5a8xlZHEDRtQQTc4
kfxENNBtVN8bE4errA06mJ10WavP2Hg+k11NQip71IQPfIF9jlk1CsqT5ZHXOrOq
RmufHFLa3fiuPvFiMB1NjK4F28Gk4LwyZrfTWc2V6S0xpL5XkFeWRW6I69xckOXj
GqkC5dsWv/IlvPeVbwIDAQAB
-----END PUBLIC KEY-----

private key :

-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQDGCglgIcCG5a8xlZHEDRtQQTc4kfxENNBtVN8bE4errA06mJ10
WavP2Hg+k11NQip71IQPfIF9jlk1CsqT5ZHXOrOqRmufHFLa3fiuPvFiMB1NjK4F
28Gk4LwyZrfTWc2V6S0xpL5XkFeWRW6I69xckOXjGqkC5dsWv/IlvPeVbwIDAQAB
AoGBAJj1yZYJm8XVg8Kdjs/Je846AOfdweYAkPfRNN2Z8RFEu5cFp5/lXtITlZRn
iAoTT/MDCtlXRkDvALH6Wstu5nvk+Xz5dFlZ6OUmK04YakHc0N5NABMygvCmGdnr
BoTJx70dPVFXqiq/ft5KRLEdxVWvex+odgWunqSJXMperrhxAkEA89/A7jadwCjA
iyaTmGEMqerN4XdTtQj4NpKorTv9FlmU9U9XWmv0wk5ExmUzjo9gs71a06/ecaHc
xJUj3X1O5wJBAM/i2zmAg1vhR/s23b70LPf4O1/d5GdQTJwHhZp4OMzW6qt7qmRR
vIzHHPbljOwvUzNtoXhez3TNsgtWg9XaXDkCQQDujYJgwoYfEP3/X9XiqZREpg2M
LjhwjvyWDXH4OwT/ltNR/rF5Hr8GTp+R3i7HldLHH0O4bIFQcD/PAABcSZYjAkAZ
QxwY1MEhvTKeGIDB37JHP+cXM0O6OkvU+iUGLG3alpNV22VNY5FiGiAu8J47ZVTa
/wuMMRlMvGJSdmT2694hAkBjqYMwpYg+MXhY9zKYM8lGISpirz+pKSpKwB90FASY
4AIuFBFHV5F5iqrSKNU39Rv+alYAKP/UuqPVH84GlWjl
-----END RSA PRIVATE KEY-----

Answer 1

A Few Comments.

1. You may find this page helpful:

   http://phpseclib.sourceforge.net/interop.html#rsaencpkcs1,p1phpseclib,p2openssl

2. I had suggested in my earlier answer that you do $rsa->setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1);. Your latest edited post still does not have you doing that:

   $rsa->loadKey($public_key); // public key
   $plaintext = '...';
   $ciphertext = $rsa->encrypt($plaintext);
   file_put_contents("ciphertext.txt",$ciphertext);
   

   Doing define('CRYPT_RSA_PKCS15_COMPAT', true); only does anything if you're in PKCS1 mode and you're not.

3. You're post says you're doing openssl rsautl -decrypt -inkey private.pem. idk how the whole xxd command is supposed to work but normally, with OpenSSL, you need to specify an input file by doing -in ciphertext.txt, which you're not doing.

Working Code

Using your public and private key this worked for me:

<?php
include('Crypt/RSA.php');

$rsa = new Crypt_RSA();

$public_key = '-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGCglgIcCG5a8xlZHEDRtQQTc4
kfxENNBtVN8bE4errA06mJ10WavP2Hg+k11NQip71IQPfIF9jlk1CsqT5ZHXOrOq
RmufHFLa3fiuPvFiMB1NjK4F28Gk4LwyZrfTWc2V6S0xpL5XkFeWRW6I69xckOXj
GqkC5dsWv/IlvPeVbwIDAQAB
-----END PUBLIC KEY-----';
$rsa->loadKey($public_key); // public key
$rsa->setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1);
$plaintext = '...';
$ciphertext = $rsa->encrypt($plaintext);
file_put_contents("ciphertext.txt",$ciphertext);

...and on the CLI:

openssl rsautl -decrypt -inkey private.pem -in ciphertext.txt

Here's the output I got:

...

Answer 2

First, phpseclib does OAEP padding by default. It's more secure but less common. Try doing $rsa->setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1);.

Second, RSA does have an upper bound on the size of data that can be encrypted. For PKCS1 it's the size of the modulo - 11 (eg. ($rsa->getSize() >> 3) - 11). idk how big the data you're trying to encrypt is but my guess is that it's bigger than that. So what phpseclib does in that situation is that it'll do str_split on it and then concatenate each of the successive ciphertext's. My guess is that that's what's happening in your case. So either you need to encrypt less data or come up with a different approach. eg. encrypt a symmetric key and then use the symmetric key to encrypt the data.