I agree with EJP's answer, but since you've accepted a different one, here are further details on what the issue with self-signed certificates are going to be.
Using a self-signed certificate on your server can be done. You'll just need to configure all the potential clients to use trust this certificate. This can be done by importing the certificate (without the private key) into each machine's trusted store, or within the browser's trusted certificate repository (for example, Firefox uses its own, independent of the OS on which it's running). Self-signed server certificates here are mainly a special case of a custom CA certificate, where each client needs to know about each server it may use (with the downside that self-signed certificate can't be revoked). This is OK, but this can quickly become a problem if you have than a few machines, in practical terms.
Your main problem will be about self-signed certificates as client certificates.
Client-certificate authentication is initiated by the server, which sends a TLS Certificate Request message to the client. This message contains a list of names of Certification Authorities which it's willing to accept. Clients then use this list to select which certificate to send: they look for certificates (for which they have the private key) that have an Issuer DN matching one of the names in this CA list (they can also use certificate chains, if intermediate CA certificates are required to make the link to an Issuer DN that's in this CA list).
Most clients will simply not send any client certificate at all if they can't find a certificate that matches these conditions. This is going to be your biggest problem when trying to use self-signed client certificates.
A way around this problem is to get the server to send an empty list. This gives more freedom to the client as to which certificate to choose (it's then up to the server to choose whether or not to accept it, but that's would always be the case anyway). This is explicitly allowed by TLS 1.1 and previous versions (SSLv3/TLS 1.0) are silent on this topic, although in practice, it works fine with most SSLv3/TLS 1.0 stacks too as far as I know. (This can still lead to clunky interactions when automatic certificate selection is done by the browser, for example, since it makes it difficult to make that automatic selection correctly.)
Java's X509TrustManager can be customised using getAcceptedIssuers() to return an empty CA list in the TLS Certificate Request message. As far as I know, in C#/.Net, SslStream's RemoteCertificateValidationCallback doesn't have its equivalent. As far as I know, this list is built from the machine's trusted certificates list when using IIS/SslStream. (Note that this is independent of the certificate verification itself, it's just about the server advertising which certificates it might accept.)
In addition, if you want to use client-certificate authentication with self-signed certificates, you'll have to implement your own verification callbacks in the server to perform this authentication. A simple example would be to extract the public key from the certificate you get, compare it against a pre-defined list the server has, and then use the user name you have in that pre-defined list. You just can't trust anything a self-signed certificate says unless you have an explicit way of checking its content against something you know. Implementing a callback that just says return true; will not perform any authentication all all. Anyone could build a certificate with the same names or attributes and different keys. (It would almost be better to trust each client-cert explicitly in your server trusted certificates store, although this may lead to long CA lists in the TLS Certificate Request message.)
If you don't want to rely on a commercial CA, build you own CA. It can be a bit of work (mostly administrative), but at least you'll have a cleaner resulting system. By configuring your clients and your servers with that CA certificate, you should be able to expand to more clients and servers without changes there, and you'll also be able to benefit from the normal CA list behaviour when the client certificate is requested.
