This is just to share my problem and solution to everyone.
I setup the plugin with the instructions from https://wiki.jenkins-ci.org/display/JENKINS/Team+Foundation+Server+Plugin
First problem: FATAL: com.microsoft.tfs.core.exceptions.TECoreException: Connection refused java.lang.RuntimeException: com.microsoft.tfs.core.exceptions.TECoreException: Connection refused
Second problem: FATAL: com.microsoft.tfs.core.exceptions.TECoreException: Unrecognized SSL message, plaintext connection? java.lang.RuntimeException: com.microsoft.tfs.core.exceptions.TECoreException: Unrecognized SSL message, plaintext connection?