Update:
Just execute:
C:\YourAgentFolderPath\agent>agent\vsoagent /unconfigure
Answer from year 2016:
Go to the folder on the Build Server where you have unzipped your build agent.
Some of the commenters mentioned, that they didn't have to delete the settings file, so you can try to skip this step, perhaps it is not necessary anymore.
Delete the settings.json file if it exists.
Open the commandline tool (cmd.exe) with admin rights!
Go to the folder as shown above in the screenshot.
Run the following command:
C:\YourAgentFolderPath\agent>agent\vsoagent /unconfigure
Result (console log):
Removing EventLog source vsoagent.vsalm.YourAgentName.
Service
vsoagent.vsalm.YourAgentName is being removed from the system...
Service
vsoagent.vsalm.YourAgentName was successfully removed from the system.
Attempt to stop service vsoagent.vsalm.YourAgentName.