I think your problm maybe to do with having 3 build agents that are not tagged. I think the workspace, if left behind, is deleted by the agent that is doing the build. If it is a different agent to the agent that created the workspace then there will be obvious issues.
So to fix the issue you need to do the following.
Name One agent the Default Agent. This will have not tags.
In the other two agents, in properties add a tag for the agents, One for each agent and select that.
Now any build that is executed that has no tag set will always use the Default Agent.
To get a build to use one of the other agents, open the build definition and goto the advanced section in Process.
Open the Agent Settings and select the ellipsis in Tags Filter and enter a tag of the same name for the the tag entered on the build agent you want to use.
You may need to clear up the workspaces before the first run.
Doing the above allows you to control what build agent is used for each build definition, and therefore should also stop your workspace issue.