CryptoAPI: Using CryptVerifySignature to verify a signature from openssl with public key

8
votes

I am trying to port the AquaticPrime framework for Mac to Windows.

On the Mac, it uses the opensll library, and I try to understand how to port this to Windows, where I have to use the CryptoAPI, I guess.

I mainly need the code for validation of the generated signature with a given public key.

Here's how verification is done with openssl:

  1. inputs: license data, public key and signature, both 128 bytes long.
  2. A SHA1 digest is calculated from the license data.
  3. A RSA context is set up with the public key data
  4. RSA_public_decrypt() is called, given the RSA key and the signature, which returns a 20 byte long SHA1 digest - is this digest equal the one from step 2, the signature is valid.

So, how do I do this with CryptoAPI? I've gotten this far:

  1. Start with CryptAcquireContext(ctx, 0, 0, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)
  2. Use CryptImportKey with the help of this posting, with pubexp=3 and bitlen=1024. That all works, i.e. I get no errors, and I looked at the binary data to verify that it matches what the MSDN article shows.
  3. Create a SHA1 digest from the license data. I've retrieved the resulting 20 byte hash value and see that it matches what I get with openssl on the Mac.

At this point, I call:

CryptVerifySignature (hashHdl, sig, sigLen, keyHdl, 0, 0)

This fails with error code ERROR_INVALID_PARAMETER.

The odd thing is that when I first accidentally had stored a twice as large public key into the PUBLICKEYBLOB structure, I received a NTE_BAD_SIGNATURE error instead. This might suggest that now the public key I am passing is correct.

Why the ERROR_INVALID_PARAMETER error now, then? I've verified that the hash value is correct, and the key appears to be accepted, too. And the "sig" parameter is just a pointer to the 128 bytes of the signature, and sigLen is 128.

So, what am I missing here?

2
opensslrsacryptoapiaquaticprime
Question: Why don't you just use some Windows ported version of OpenSSL? I mean there's nothing wrong in using CryptoAPI but openSSL already exists and it would mean less work... – erloewe
I don't want to require people to install new system components just to run an app that doesn't even require an installer otherwise. – Thomas Tempelmann

2 Answers

9
votes

OK, I solved the problem after lots of trial-and-error.

Both the signature and the public key data, when in their pure byte string form, need to be reversed, i.e. first byte to last position, and so on. Then the above works.

-3
votes

Compile and link the OpenSSL libCrypto statically. It can be done, I've seen this at a former employer.