Manual validation Lotus Notes' signature with public key

Question

We have Lotus Notes signed document and user's public key. What we need to do: enter the key into a field in a special application (it can be Lotus notes database or some special soft). Then we ask this special application: "Is this document really signed by this user with this public key?" And our app must answer: yes or not.

We try to write this special application and we met few issues:

1. We have field named $Signature in the document, that is the hash of signed fields, encrypted with private key of the signer. I can see content of this field in document's properties. But I can't to extract it programmatically (I tried LotusScript and Java). And I didn't find any way to do it.

2. Therefore I just manually copied content of this field and pasted it into a field on a special form to further analyze. But there I met another problem. I don't know how to decrypt this signature. What algorithm Lotus uses to sign hash? If I will know the algorithm I guess I will be able to decrypt it with Java and get hash of signed fields.

3. And there will be one more problem I believe. I dont know how Lotus counts hash of fields. Does it use md5? I need to know it to be able to compare hashes and say did this user signed the document or not.

So. It's the interest task. But now it's impossible to solve it. There are 3 huge problems on the way. Can anyone help with them?

Answer 1

The answer is: don't try to do this yourself. Not the way you described it. There's an API to validate Notes signatures.

Just copy the document's UNID to your database, and then write code using the Notes C API to open the document and call the API function NSFNoteVerifySignature() to validate it. You can do this from Java using JNI or from LotusScript by following the techniques that are described here, or you can use the LSX toolkit, or just write a standalone C program.

You would have to use the Notes C API anyhow to deal with two of the three points that your raised:

1. You need the C API to get get at the contents of the $Signature item.
2. The signature is RSA.
3. You actually have two problems: the algorithm, and the input. You have to match them both. If I recall correctly, Lotus has described the hash algorithm as "modified MD2". Bear in mind, this goes back well over 20 years, and breaking compatibility is something that they don't like to do. It's possible that they've changed it when they upgraded RSA key sizes, but I don't recall hearing about that. But as I said, that's only half the problem. You need to get the raw input bytes in exactly the same format as the signature algorithm saw them, and for rich text fields this probably means reading raw CD records, which requires the C API.