Security in WCF actually consists of several features. The difference between those two is how are messages signed and encrypted.
Transport security provides only point-to-point channel security. It means that HTTPS establish secure channel only between client and server exposed to client. But if this server is just a load balancer or reverse proxy server it has direct access to content of the message.
Message security provides end-to-end channel security. It means that security is part of transferred data and only intended destination can decrypt the data (load balancer or proxy sees only encrypted message). Message security in most cases also uses certificates to provide encryption and signing but it is usually slower because transport security can use HW acceleration.
In advanced scenarios these methods can be combined. For example you can have communication to your load balancer secured by HTTPS because you trust your internal network after load balancer but in the same time you can have the message signed (message security) so you can prove that it wasn't changed.
Another difference between those two is that transport security is related to single transport protocol whereas message security is independent on transport protocol.
Message security is based on interoperable protocols (but be aware that not every configuration in WCF is interoperable). WCF supports at least partially these protocols:
- WS-Security 1.0 and 1.1 - basic rules for encryption, signing, token transport, timestamps, etc.
- UserName token profile 1.0 - definition of token used for transporting user name and password. This specification is implemented only partially because WCF out of the box doesn't support digested password and requires using this token either with transport or message encryption.
- X509 token profile 1.1 - definition of token used for transporting certificates.
- Kerberos token profile 1.1 - definition of token used for transporting Kerberos tickets.
- SAML 1.1 token profile 1.0 and 1.1 - definition of token used for federated security. SAML 2.0 is provided by WIF.
- WS-SecurityPolicy 1.1 and 1.2 - provides support for defining security assertion in WSDL.
- WS-SecureConversation 1.3 and Feb. 2005 - provides support for security session where credentials are exchanged only during first call and rest of the communication uses unique security token.
- WS-Trust 1.3 and Feb. 2005 - provides support for federated scenarios and Security token services (STS).
WCF also supports WS-I Basic Security Profile 1.0 which is just subset of former protocols with prescribed configuration.
For non interoperable features WCF offers features like Windows security or TLSNego and SPNego (both should be generally interoperable but their are not available in many SOAP stacks) for service credentials exchange.