Digital signature generated Java will not verify in openssl

Question

I am signing a binary file in Java. I am able to sign and verify within the Java code, however when using the generated signature in openssl it fails to verify to the same binary. The steps are: Load data -> create SHA256 hash -> sign the hash

This should be the equivalent of doing this in openssl: openssl dgst -sha256 -sign private.key -binary target.bin > signature.bin verify: openssl dgst -sha256 -verify public.key -signature signature.bin

I verified the sha256 matches the one generated from openssl.

public class Main {

    public static void main(final String[] args) {
        Signature signer;
        KeyStore ks;
        byte[] signature;
        PrivateKey privateKey;
        PublicKey publicKey;
        String alias = "ctest3";
        String sigfile = "c:\\temp\\signature";
        String datafile = "c:\\temp\\data.bin";

        try {
            ks = KeyStore.getInstance("ncipher.sworld", "nCipherKM");
            FileInputStream in = new FileInputStream("C:\\private\\ctest3.ncsw");
            ks.load(in, null);
        } catch (KeyStoreException |
                NoSuchAlgorithmException |
                CertificateException |
                IOException |
                NoSuchProviderException e) {
            System.err.println(e.getMessage());
            return;
        }

        try {
            privateKey = (PrivateKey) ks.getKey(alias, null);
            Certificate cert = ks.getCertificate(alias);
            publicKey = cert.getPublicKey();
        } catch (Exception e) {
            System.out.println(e.getMessage());
            return;
        }

        byte[] payload;
        try {
            payload = Files.readAllBytes(Paths.get(datafile));
        } catch (IOException e) {
            System.out.println(e.getMessage());
            return;
        }

        byte[] spayload = sha256(payload, false);
        if (spayload == null) {
            return;
        }

        try {
            signer = Signature.getInstance("SHA256withECDSA");
            signer.initSign(privateKey);
            signer.update(spayload);
            signature = signer.sign();
        } catch (NoSuchAlgorithmException | SecurityException | SignatureException | InvalidKeyException e) {
            System.out.println(e.getMessage());
            return;
        }

        writeBytesToFile(signature, sigfile);
        KeyPair kp = new KeyPair(publicKey, privateKey);

        if (Verify(datafile, sigfile, kp)) {
            System.out.println("success");
        }
    }

    private static byte[] sha256(byte[] data) {
        MessageDigest digest;
        try {
            digest = MessageDigest.getInstance("SHA-256");
            byte[] mDigest = digest.digest(data);
            return mDigest;
        } catch (NoSuchAlgorithmException e) {
            return null;
        }
    }

    private static boolean Verify(String filename, String signaturePath, KeyPair kp) {
        Signature verify;
        byte[] dataRaw;
        byte[] signature;
        byte[] data;

        try {
            dataRaw = Files.readAllBytes(Paths.get(filename));
            signature = Files.readAllBytes(Paths.get(signaturePath));
        } catch (IOException e) {
            System.out.println(e.getMessage());
            return false;
        }

        data = sha256(dataRaw, false);
        if (data == null)
            return false;

        try {
            verify = Signature.getInstance("SHA256withECDSA");
            PublicKey pub = kp.getPublic();
            verify.initVerify(pub);
            verify.update(data);
            if (verify.verify(signature))
                return true;
        } catch (NoSuchAlgorithmException | SecurityException | SignatureException | InvalidKeyException e) {
            System.out.println(e.getMessage());
            return false;
        }

        return false;
    }

    private static void writeBytesToFile(byte[] bFile, String fileDest) {
        try (FileOutputStream fileOuputStream = new FileOutputStream(fileDest)) {
            fileOuputStream.write(bFile);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Robert Daniels Robert Daniels · Accepted Answer · 2019-08-26T22:08:12

Turns out the Signature object, when initiated with SHA256withECDSA keys, will itself generate the SHA256 hash during the sign() call. Removing the call to "sha256()" in the above code verifies with OpenSSL.

Digital signature generated Java will not verify in openssl

1 Answers