Option 1: You need to adds .tfignore to the packages folder with "*/" as a filter. You effectively don't ever want to check anything in under this folder other than the config file.
http://msdn.microsoft.com/en-us/library/ms245454.aspx
Option 2: (Better) Add a nuget.config file to your solution, convention is to place it under a folder named .nuget - but it can stay at root too. Content should be:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<solution>
<add key="disableSourceControlIntegration" value="true" />
</solution>
</configuration>
That will exclude the packages folder, you don't need anything in there. NOTE: If you're using NuGet 2.7 or above, automatic restore is on by default. Don't use the "Enable NuGet Restore" in your solution, it adds the old way of doing this. See http://geekswithblogs.net/terje/archive/2014/06/11/converting-projects-to-use-automatic-nuget-restore.aspx