When you build a project the .xml/.pdb files are gathered through the ResolveAssemblyReference task. When ResolveAssemblyReference is called it is passed a list of file extensions for related files. That list of file extensions is captured in the MSBuild property AllowedReferenceRelatedFileExtensions. By default that list will contain ".pdb;.xml".
If you want to exclude all related reference files from being picked up then just override the value of the property to something which related files won't have extensions of. For example you can set AllowedReferenceRelatedFileExtensions to "-".
You can also customize the list of file which are returned by that. If you only want to find only .pdb files then you will need to pass in AllowedReferenceRelatedFileExtensions=".pdb"
. In that case any references which have .pdb file next to the .dll/.exe they will be copied as well. You can also use this to copy other related files which may not end in .pdb/.xml. For example if you have a referenced assembly named, MyAssembly.dll and in that same folder there exists MyAssembly.pdb and MyAssembly.foo If you set AllowedReferenceRelatedFileExtensions=".pdb;.foo"
then both the .pdb and .foo file will be copied to the output directory.