Service Fabric Local Cluster fails to get certificate(s) private key(s)

Question

For every Service Fabric application I attempt to run which utilizes one or more SecretsCertificate instances, the application fails to launch in my local Service Fabric cluster with the following error on the Node > Application in the SF Explorer:

• Error event: SourceId='System.Hosting', Property='Activation:1.0'. There was an error during activation.Failed to configure certificate permissions. Error E_FAIL.

Service Fabric also logs a few relevant items in to the Event Viewer > Applications and Services Logs > Microsoft-Service Fabric > Admin section:

• CryptAcquireCertificatePrivateKey failed. Error:0x8009200b
• Can't get private key filename for certificate. Error: 0x8009200b
• All tries to get private key filename failed.
• Failed to get the Certificate's private key.
• Thumbprint:4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXC. Error: E_FAIL
• Failed to get private key file. x509FindValue: 4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXC, x509StoreName: My, findType: FindByThumbprint, Error E_FAIL
• ACLing private key filename for thumbprint 4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXC. ErrorCode=E_FAIL
• ConfigureCertificateACLs: error=E_FAIL

I have removed and reinstalled the certificate (which is confirmed to work in multiple other developers' local Service Fabric cluster development environments), and set the private key to have explicit full control permissions for the NETWORK SERVICE user on my computer, which didn't help.

I have followed the instructions in this answer which actually prints out the private key details correctly despite SF local cluster not being able to access it.

I have reinstalled Microsoft Service Fabric SDK, and Microsoft Visual Studio 2017 which also didn't resolve this problem.

All attempts to recreate this error in C# and PowerShell have been fruitless, yet the Service Fabric service doesn't seem to be able to access private keys from my cert store.

Edit: Further progress, no solution.

I am able to successfully decrypt data using the PowerShell Invoke-ServiceFabricDecryptText cmdlet, yet the SF Local Cluster still has the same error.

I determined that the file specified in the certificate's metadata (from the previously referenced SO answer) PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName doesn't exist on my disk at the path C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys\ or any neighboring paths. Has anyone seen this before?

Answer 1

As discussed in the comments, the issue is related to how the (self-signed) certificate is created. When using Powershell to create your certs make sure to use:

So when I specified -Provider "Microsoft Enhanced Cryptographic Provider v1.0" for the New-SelfsignedCertificate command to create a cert, it works.

Source: https://github.com/Azure/service-fabric-issues/issues/235#issuecomment-292667379

Answer 2

An alternative, in case you can't or don't want to use a self-signed certificate, is to "remove" the CNG storage of the private key (which is the part that Service Fabric can't yet handle).

The steps outlined in this article show how to convert a CNG cert to a non-CNG one: https://blog.davidchristiansen.com/2016/05/521/

1. Extract your public key and full certificate chain from your PFX file

openssl pkcs12 -in "yourcertificate.pfx" -nokeys -out "yourcertificate.cer" 
    -passin "pass:password"

2. Extract the CNG private key

openssl pkcs12 -in "yourcertificate.pfx" -nocerts –out “yourcertificate.pem"
    -nodes -passin "pass:password" -passout "pass:password"

3. Convert the private key to RSA format

openssl rsa -inform PEM -in "yourcertificate.pem" -out "yourcertificate.rsa"
    -passin "pass:password" -passout "pass:password"

4. Merge public keys with RSA private key to a new PFX file

openssl pkcs12 -export -in "yourcertificate.cer" -inkey "yourcertificate.rsa" 
    -out "yourcertificate-converted.pfx" 
    -passin "pass:password" -passout "pass:password"