Azure Cloud Service (Classic) - single instance is missing a root CA / SSL certificate cannot be verified

Question

I have a Cloud Service (Classic) running in Azure that hosts a WCF service. Multiple instances are configured in the Cloud Service thorugh the Scale feature of the Cloud Service. An SSL Wildcard certificate has been provided via Azure Portal and configured in csdef/cscfg in the corresponding Visual Studio Cloud project. Until recently, everything run perfectly, no issues with the certificates, etc. has occurred.

.csdef

<Sites>
  <Site name="Web">
    <Bindings>
      <Binding name="Endpoint2" endpointName="EndpointSSL" />
    </Bindings>
  </Site>
</Sites>
<Endpoints>
  <InputEndpoint name="EndpointSSL" protocol="https" port="443" certificate="SSL" />
</Endpoints>
<Certificates>
  <Certificate name="SSL" storeLocation="LocalMachine" storeName="My" />
</Certificates>

.cscfg

<Role name="FR">
  <Instances count="3" />
  <ConfigurationSettings>
    <!-- Settings are set here -->
  </ConfigurationSettings>
  <Certificates>
    <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption" thumbprint="THUMBPRINT_A" thumbprintAlgorithm="sha1" />
    <Certificate name="SSL" thumbprint="THUMBPRINT_B" thumbprintAlgorithm="sha1" />      
  </Certificates>
</Role>

The certificate gets installed properly, as you can see in the certificate management window in windows. Unfortunately, starting last week, when checking the site via SSL Labs (https://www.ssllabs.com/ssltest/) it started to deliver two certificates in the resulting report, one of them being invalid as the certificate chain seems to be broken. Other services using the same Wildcard certificate do not have this issue.

What I do know so far about this issue:

• The certificate in the certificate store on one of the three running instances cannot be verifies by windows itself.
• On the other two instances, the certificate is correct and can be verified by windows
• The instance with the invalid certificate does not have the corresponding root CA in the "Trusted Root CAs" store in windows.
• Redeploying, Reimaging, Restarting, etc. does not help. It might just shift the issue from one instance to another. SO far only one instance was affected at a time.
• No changes have been made manually nor was a newer deployment applied recently. It just started to occur out of the blue

Does anybody have any ideas what the issue might be? I also started a support ticket with Microsoft but so far they also have no clue. Are there any settings that I might try out when deploying that forces the certificates to be verified at deployment? Or is there any option to get the missing Root CA from code at startup (as I suspect that this might be the issue why it cannot be verified on the one instance)?

Answer 1

The Microsoft Azure support does also not know the root cause of this behavior, however they came up with a suitable workaround for the time being.

In the Cloud Service, a Startup Task has been added that checks whether the potentially missing Root CA is existing in the list of all Root CAs. If not, the missing one will be installed (via Powershell). After that, the missing Root CA should have been added to the list of Root CAs and the actual SSL Wildcard certificate (that failed to validate) can successfully be verified - checks via SSLLabs will work again.

At least for my problem, this did the trick. I'm not completely satisfied with it as I'd like to know the root cause, but Microsoft seems to be aware of the issue and (from what I understand) is investigating investigating further. If any proper solution (instead of the workaround) comes up, I will try to add it here as well.

CMD File for Startup task:

SET LOG_FILE="%TEMP%\StartupLog.txt"
SET EXECUTE_PS1=0

IF "%ComputeEmulatorRunning%" == "" (
    SET EXECUTE_PS1=1
)

IF "%ComputeEmulatorRunning%" == "false" (
    SET EXECUTE_PS1=1
) 

IF %EXECUTE_PS1% EQU 1 (
    echo "Invoking InstallCertificateSSLConfigure.ps1 on Azure service at %TIME% on %DATE%" >> %LOG_FILE% 2>&1  
    PowerShell -ExecutionPolicy Unrestricted .\Startup\InstallCertificate.ps1 >> %LOG_FILE% 2>&1
    IF %ERRORLEVEL% NEQ 0 (EXIT /B %ERRORLEVEL%)
) ELSE (
    echo "Skipping InstallCertificate.ps1 invocation on emulated environment" >> %LOG_FILE% 2>&1    
)    

EXIT /B 0 

Powershell script to install certificate (executed from CMD)

$thumbprint = "THUMBPRINT"
$path = ".\Startup\MISSING ROOT CA.crt"

$cert = Get-ChildItem -Path Cert:\LocalMachine\Root | Where-Object {$_.Thumbprint -eq $thumbprint}

if ( $cert )
{
    Write-Host "The Certificate $($cert.Thumbprint) is installed already"
}
else
{
    Write-Host "The Certificate $thumbprint is not installed"
    Import-Certificate -FilePath $path -CertStoreLocation Cert:\LocalMachine\Root
    Write-Host "installed the Certificate $thumbprint"
}