I didn't have much luck with the other answers, I finally figured out how to do this in my implementation by using the built in "Delete" command, apparently there is a specific way you need to implement wildcards, it's bit nuanced, here's everything you need to be put into your "CSPROJ" (TargetDir is a built in variable, included automatically) under the "Project" tag:
<Target Name="RemoveFilesAfterBuild">
<ItemGroup>
<XMLFilesToDelete Include="$(TargetDir)\*.xml"/>
<PDBFilesToDelete Include="$(TargetDir)\*.pdb"/>
</ItemGroup>
<Delete Files="@(XMLFilesToDelete)" />
<Delete Files="@(PDBFilesToDelete)" />
</Target>
I've also had trouble with various language specific folders being generated, if you have that issue too, you can also remove unused language specific folders too. I've chosen to only trigger this under the build type "Release":
<ItemGroup>
<FluentValidationExcludedCultures Include="be;cs;cs-CZ;da;de;es;fa;fi;fr;ja;it;ko;mk;nl;pl;pt;ru;sv;tr;uk;zh-CN;zh-CHS;zh-CHT">
<InProject>false</InProject>
</FluentValidationExcludedCultures>
</ItemGroup>
<Target Name="RemoveTranslationsAfterBuild" AfterTargets="AfterBuild" Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<RemoveDir Directories="@(FluentValidationExcludedCultures->'$(OutputPath)%(Filename)')" />
<ItemGroup>
<XMLFilesToDelete Include="$(TargetDir)\*.xml"/>
<PDBFilesToDelete Include="$(TargetDir)\*.pdb"/>
</ItemGroup>
<Delete Files="@(XMLFilesToDelete)" />
<Delete Files="@(PDBFilesToDelete)" />
</Target>