I just finished setup two build agent on a server. From what I think, I don't need to install TFS Server on this machine because it use as build agents not the server itself. I might need to install TFS Server to configure the build agents but I hope I can uninstall them after I finished configuring.
I am not sure is it possible to run TFS Build Agent without installing TFS server or not?
You do not have to have the application tier (server) configured on the build agent machine.
You can install the build agent but in order to fully configure the build agent, it will need to be configured against a server's collection (local or remote).
Step 3 in the wizard covered here, details how you need to browse and select the collection the build agent will build for by browsing (it's the same connect dialog you get from the VS client).
Note that if you are using TFS 2012 Build Wizard on-premise (it's released now) it saves you from deploying a server :) At step 3, you point to a cloud hosted account @ tfspreview.com, it will prompt you to authenticate and download the service account details for the build agent.
The 2012 wizard is covered here.
