The issues I had was with no support of RSA-SHA512 signature algorithm. It can be achieved by using Custom Signature Algorithm. Then add it to Default Signature Algorithm in your custom binding.
public class MyCustomAlgorithmSuite : SecurityAlgorithmSuite { }
Also create Signature for this as:
public class RsaPkCs1Sha512SignatureDescription : SignatureDescription
{}
Then add it to :
CryptoConfig.AddAlgorithm(typeof(RsaPkCs1Sha512SignatureDescription),
"http://www.w3.org/2001/04/xmldsig-more#rsa-sha512");
However we ran into an issue with signing BinarySecurity Token. It can be signed by adding:
assymetricKey.EndpointSupportingTokenParameters.Signed.Add(new X509SecurityTokenParameters());
However this creates a new signed Binary Security Token and you will end up with two BST. The solution to this is by :
assymetricKey.InitiatorTokenParameters = new System.ServiceModel.Security.Tokens.X509SecurityTokenParameters { InclusionMode = SecurityTokenInclusionMode.Never};
assymetricKey.RecipientTokenParameters = new System.ServiceModel.Security.Tokens.X509SecurityTokenParameters { InclusionMode = SecurityTokenInclusionMode.Never};
This messes up with your SignedInfo to lose reference to BST and we are stuck.
Anyway we ended up not using WCF capability and instead used:
SignedXML class.
For reference to how to use it here is the link:
https://gist.github.com/luizvaz/43ccbd85b16b6802218b50b6d34c26de
Also, thing to note here is if you are implementing signature algorithm RSA-SHA512 from certificate then you will need to use extension method (make sure to use .net 4.6.2 or higher). For old versions:
SignedXml Compute Signature with SHA256
signedXml.SigningKey = RSACertificateExtensions.GetRSAPrivateKey(cert);
Also, if you are adding prefix to the Signature as ds
then you will need to remove references from SignedInfo
, re-compute signature and add it back.
Generate Digital Signature but with a Specific Namespace Prefix ("ds:")
I hope this helps to having similar issues.