Yes, you can set permission on TFVC up to a single file; for Git version control in TFS, you can control access to a repository or at most to a branch.
That said, it is correct to ask yourself if it is worthwhile, as every time you break inheritance on security you face more administrative work in the future. The leaner and simpler the configuration, the less work for the administrator.
Some scenarios where this make sense are:
- a big project space with multiple teams (read carefully: not a big project, but a single TFS Project spanning many loosely related sub-projects)
- a hosting configuration, which means just the same as above
- protecting some special files, so they are accessible to special accounts, like certificates, private keys, etc.