Unable to do RSA Encryption/Decryption using Crypto++ (isValidCoding is false)

Question

I am using Crypto++ to encrypt an array of bytes using RSA. I have followed Crypto++ wiki's samples with no luck getting them to work. Encryption and Decryption in all the samples are done within a single process but I am trying to decrypt the content which is already encrypted in another process. Here is my code:

class FixedRNG : public CryptoPP::RandomNumberGenerator
{
public:
    FixedRNG(CryptoPP::BufferedTransformation &source) : m_source(source) {}

    void GenerateBlock(byte *output, size_t size)
    {
        m_source.Get(output, size);
    }

private:
    CryptoPP::BufferedTransformation &m_source;
};


uint16_t Encrypt()
{
    byte *oaepSeed = new byte[2048];
    for (int i =  0; i < 2048; i++)
    {
        oaepSeed[i] = (byte)i;
    }
    CryptoPP::ByteQueue bq;
    bq.Put(oaepSeed, 2048);
    FixedRNG prng(bq);
    Integer n("Value of N"),
    e("11H"),
    d("Value of D");
    RSA::PrivateKey privKey;
    privKey.Initialize(n, e, d);
    RSA::PublicKey pubKey(privKey);
    CryptoPP::RSAES_OAEP_SHA_Encryptor encryptor( pubKey );
    assert( 0 != encryptor.FixedMaxPlaintextLength() );
    byte blockSize = encryptor.FixedMaxPlaintextLength();
    int divisionCount = fileSize / blockSize;
    int proccessedBytes = 0;
    // Create cipher text space
    uint16_t cipherSize = encryptor.CiphertextLength( blockSize );
    assert( 0 != cipherSize );

    encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
    return cipherSize;
}

void Decrypt(uint16_t cipherSize)
{
        byte *oaepSeed = new byte[2048];
        for (int i =  0; i < 2048; i++)
        {
            oaepSeed[i] = (byte)i;
        }
        CryptoPP::ByteQueue bq;
        bq.Put(oaepSeed, 2048);
        FixedRNG prng(bq);
        Integer n("Value of N"),
        e("11H"),
        d("Value of D");
        RSA::PrivateKey privKey;
        privKey.Initialize(n, e, d);
        //RSA::PublicKey pubKey(privKey);

        CryptoPP::RSAES_OAEP_SHA_Decryptor decryptor( privKey );

        byte blockSize = decryptor.FixedMaxPlaintextLength();
        assert(blockSize != 0);

        size_t maxPlainTextSize = decryptor.MaxPlaintextLength( cipherSize );
        assert( 0 != maxPlainTextSize );
        void* subBuffer = malloc(maxPlainTextSize);
        CryptoPP::DecodingResult result = decryptor.Decrypt(prng, (byte*)cipherText, cipherSize, (byte*)subBuffer);
        assert( result.isValidCoding );
        assert( result.messageLength <= maxPlainTextSize );
}

Unfortunately, value of isValidCoding is false. I think I am misunderstanding something about RSA encryption/decryption!!
Note that, privKey and pubKey have been validated using KEY.Validate(prng, 3). I have also tried to use RAW RSA instead of OAEP and SHA with no luck. I have tried to debug through crypto++ code, what I am suspicious about is prng variable. I think there is something wrong with it. I have also used AutoSeededRandomPool instead of FixedRNG but it didn't help.
Worth to know that, if I copy the decryption code right after encryption code and execute it in Encrypt() method, everything is fine and isValidCoding is true!!

You need to debug this more closely. I can't find any direct fault with the code. Why void* subBuffer = malloc(maxPlainTextSize); — Captain Giraffe
Dear @CaptainGiraffe, Thanks for your comment. I've been debugging this more than one week. I usually hesitate to ask a question on StackOverflow unless I have tried everything that I could think of. In RSA decryption we don't know the exact length of plainttext before decryption is done, therefore we have to create a buffer with the size of decryptor.MaxPlaintextLength(cipherSize) and then trim it to result.messageLength. The most annoying thing about the code is that it successfully works when I do copy and paste it right after the place where encryption takes place!! — Farzan
Here's a reference you might want to look at: cryptopp.com/wiki/RSA_Encryption_Schemes. — jww
@noloader thanks for your comment. Well I didn't say the code is a copy-paste from the wiki, I did modify the examples to match my requirements. I didn't include value of n and d in this question but to be precise I am using 384bit values. I am using if(!publicKey.Validate(prng, 3)) to throw an error if the key is not validated. Thanks for the NullRNG suggestion, I'll give it a try. I'll also follow examples in your given link. — Farzan

jww jww · Accepted Answer · 2014-04-17T06:42:11

This is probably not be correct:

byte blockSize = encryptor.FixedMaxPlaintextLength();
...

encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
return cipherSize;

Try:

size_t maxLength = encryptor.FixedMaxPlaintextLength();
size_t cipherLength = encryptor.CiphertextLength( blockSize );
...

SecureByteBlock secBlock(cipherLength);

cipherLength = encryptor.Encrypt(prng, (byte*)plaintext, blockSize, secBlock);
secBlock.resize(cipherLength);

FixedMaxPlaintextLength returns a size_t, not a byte.

You should probably be calling CiphertextLength on plaintext.

I'm not really sure how you are just returning an uint_t from encrypt().

You might do better by starting fresh, and using an example from the Crypto++ as a starting point. I'm not sure this design is worth pursuing.

If you start over, then Shoup's Elliptic Curve Integrated Encryption Scheme (ECIES) would be a good choice since it combines public key with symmetric ciphers and authentication tags.

Unable to do RSA Encryption/Decryption using Crypto++ (isValidCoding is false)

1 Answers