0
votes

Today I started trying out the Cryptopp library. I took an example from the documentation here.

The program crashes and I cannot seem to figure out why.

My code:

#include <cryptopp/dll.h>
#include <cryptopp/hex.h>
#include <string>

int main(int argc, const char* argv[]) {

    std::cout << "START OF PROGRAM" << std::endl;

    {
        std::string encoded;
        CryptoPP::byte decoded[] = { 0xFF, 0xEE, 0xDD, 0xCC, 0xBB, 0xAA, 0x99, 0x88, 0x77, 0x66, 0x55, 0x44, 0x33, 0x22, 0x11, 0x00 };

        //Method 1
        CryptoPP::HexEncoder encoder2;
        encoder2.Attach(new CryptoPP::StringSink(encoded));
        encoder2.Put(decoded, sizeof(decoded));
        encoder2.MessageEnd();

        //Method 2
        //CryptoPP::StringSource ss(decoded, sizeof(decoded), true, new CryptoPP::HexEncoder(new CryptoPP::StringSink(encoded)));

        std::cout << "Encoded: ";
        std::cout << encoded;
        std::cout << std::endl;
    }

    std::cout << "END OF PROGRAM" << std::endl;
}

The output is:

START OF PROGRAM
Encoded: FFEEDDCCBBAA99887766554433221100

Which means that END OF PROGRAMis not reached, but the HexEncoder did its job as expected. As soon as my std::string encoded; goes out of scope the program crashes. If I move std::string encoded; to the outer scope (the scope of int main(...)) then END OF PROGRAM is printed and the program crashes at the end of main.

Also, if I don't call encoder2.Put(...) the program does not crash, but obviously the HexEncoder also does nothing. Using Method 2 instead of Method 1 results in the same error.

The error I get is:

HEAP[app.exe]: Invalid address specified to RtlValidateHeap( 000001CE84480000, 000001CE84761390 )

About my environment:

I use Microsoft Visual Studio 2019. I used the VS-solution in the cryptopp source to build the library using the DLL-Import Debug build-config. The changed I made to the config are:

  • I had to change the Runtime Library setting from /MTd to /MDd (since the larger program I'm working on and want to use cryptopp in is using /MDd).

  • I also removed the CRYPTOPP_ENABLE_COMPLIANCE_WITH_FIPS_140_2=1 preprocessor-definition since it caused issues (I think because of running my program in the Visual Studio debugger). I don't care about FIPS-compliance at this point.

Other details

Another example which involved FileSink which takes a stringstream did work as expected without the issue I described above:

std::stringstream encoded;
CryptoPP::HexEncoder encoder2(new CryptoPP::FileSink(encoded));
encoder2.Put(decoded, sizeof(decoded));
encoder2.MessageEnd();
std::cout << encoded.str();

And some testing I did involving symmetric Salsa20 encryption and decryption (example from here) did also work properly (building and running).

Question

Can somebody explain what goes wrong, how I can fix it or if it is a bug in the library or my building process?

std::string encoded; -- Reserve space before using the string: encoded.reserve(1000);. Does your program crash? If not, then this maybe an issue of the string using different heaps in different modules.PaulMcKenzie
resize will increase the size of the string and fill it with the default character. reserve will allocate memory but not fill it (i.e.: free to put some data on it).JuanR
I think it is not included because they do it in the code. But the condition is wrong: if (length < size && size + length > m_output->capacity()) m_output->reserve(2*size); They will resize the string not only if the current size plus the length is greater than the capacity, but also if the length is less than size, which doesn't look well. I would report it as a bug.JuanR
Given that reserve() worked, it seems to be an issue of allocating memory (via adding characters to the string) in one module from one heap, and then when the string is destroyed, it is being destroyed in the "source" module, which is using a different heap. Thus the run-time error. Make sure you are building all your components with the same compiler options, runtime options, and if this is Windows, you're using the Multithreaded DLL runtime, so that all objects that cross module boundaries use the same heap.PaulMcKenzie
BTW, the reserve() trick worked, since as soon as the string was created in module A, you reserved all the memory it needed in module A. Thus module B (which I guess is where the crypto functions reside) will have no reason to fool around with the heap for the string, since the string has the memory already reserved for it. Originally, you created an empty string, and thus the crypto module will need to reserve memory, but it's going to be module A that will be destroying the string, which will be looking at an "alien" heap allocated buffer. This is all speculation, but a plausible guess.PaulMcKenzie