Verifying Authenticode signed executables and DLLs using OpenSSL API

5
votes

I have installed openssl and now the rsa_test.c is running fine. What I want to do is:

  1. Open any exe or dll digital certificate. Extract the Thumbprint and PublicKey. The public key contains the exponent too (if you are familiar with ASN1 encoding).

  2. Now I have already calculated SHA-1 hash of the file and compared it to the MD field to find it correct. So now I want to pass the Thumbprint, PublicKey->m and PublicKey->e to RSA_public_decrypt function to calculate the decrypted SHA-1 hash of the file.

What Padding Algorithm should I pass to the function? When I pass RSA_NO_PADDING, it decrypts but give me the 128-bytes long hash which is not at all correct. For all other padding algorithm options it returns the error of padding not found.

Would you please teach me or tell me the correct parameters to pass through the function with an example or good link to example?

1
ccryptographyopensslrsaauthenticode
nice question: I suggest also post you question at security.stackexchange.com... – Grijesh Chauhan
Could you just post such a 128 byte block decrypted RSA signature? – Robert
@WhozCraig- I did it manually as per the code given in RFC. But I have removed the corresponding regions as explained in Authenticode Documentation. The hash is correct as I verified it with the MD field of the certificate. – user1696837
Good question: I think this is the most common problem encountered everyday by lots of coders who try using OpenSSL to verify the signature. – Abhineet
in man for RSA_public_decrypt it's stated: "This function does not handle the algorithmIdentifier specified in PKCS #1. When generating or verifying PKCS #1 signatures, rsa_sign(3) and rsa_verify(3) should be used." If you still want to use the decrypt function (why?) be sure that you have correct endianness for both m, e, and the encrypted digest. – Rostislav Kondratenko

1 Answers

3
votes

You should use the PKCS7_verify function to verify the whole PKCS #7 signedData block.

You need two steps to completely verify the signature (see Microsoft specs):

  1. Verify the integrity and identity of the PKCS #7 signature.
  2. Calculate the Message Digest and compare it to the one specified in the signature.

Note that the digest is not limited to SHA-1 in the specs. The digest algorithm is contained in digestAlgorithm field in the signature.