HI there. Since no else has answered, I'll take a crack at this - but fair warning - I'm a Java/UNIX developer, and some of what you're asking is Microsoft specific. But here's a few answers:
1 - Issue a certificate with
CN=app-server-NLB-hostname and
'Server authentication' purpose in
the domain CA.
The purpose is a somewhat Microsoft specific thing. This sounds right, but also - you may need the regular "key usage" to be something particular - for SSL certs I ususally use keyEncipherment or keyAgreement. Sometimes sites use digitalSignature. All three are valid in the RFC, but sometimes Microsoft is wierd about what workds. If you're using a Microsoft CA, I'd see if it has a certificate profile for SSL certs and just use that.
2 - Issue a certificate for front-end
boxes with 'Client authentication'
purpose.
Same as #1 - every certificate needs some sort of basic key usage. The fields you're mentioning are extended usages, which are probably also necessary, but not mandatory.
3 - Import back-end certificate's public
key into front-end nodes' storages
(and vice versa).
Only if this is a microsoft specific thing. In proper PKI, you should import the trusted CA's certificate that signed the SSL, but you should NOT have to import every end point (like the back end cert) into the front end. I would try it with only configuring your trusted CA chains and see if you can get it to work - it will make your architecture far more extensible in the long run.
4 - Configure WCF to use net.tpc with
transport security
5 - Configure the service behavior
(serviceCredentials section)
6 - Configure the client endpoint
behavior (clientCredentials section)
Sounds good to me...
From there, I don't think you've missed anything else. The key is usually working through the obscure error messages. Every system has it's own inscrutable way of telling what's wrong when an SSL handshake fails, so it's mostly getting a sense of what's gone wrong.
From the problem you describe, I don't think your security architecture is saying you need to do a certificate status check. That's an add on measure that would let your system check on whether a certificate has been revoked by the CA. It takes quite a bit of infrastruture to make it work (CRLs or OCSP), so I wouldn't say you should turn go for it without serious discussion in your team about what risks you're trying to mitigate and how likely they are to occur.
Which hostname frontend certificates should be issued for?
The hostname of the server they represent.
A thing about host names - some systems will not work properly unless you use the fully qualified domain name. Others are not so picky. In any case - the certificate's Subject DN should uniquely describe the service, application or server it represents.
The front-end nodes are in DMZ so will have no access to the domain (CA). Will this cause any problems?
You will not have problems.
UNLESS -- you need to do the aforementioned certificate status check. If you have to validate certificate status, then you need to get from the end point (ie, the DMZ) to the information about the certificate's status (the CRL or the OCSP server). In complex infrastructures, this is usually done by opening a path to the OCSP server - it's an HTTP GET to a fixed domain name or IP, so it's usually not too bad to open a path on a firewall.
Like I said- that would be the upscale, serious risk mitigation type solution. It could be overkill for your system - I don't know enough about your system to tell you whether it's warranted or not.
