(pip maintainer here!)
The specific issue of "installing the wrong version due to caching" issue mentioned in the question was fixed in pip 1.4 (back in 2013!):
Fix a number of issues related to cleaning up and not reusing build directories. (#413, #709, #634, #602, #939, #865, #948)
Since pip 6.0 (back in 2014!), pip install, pip download and pip wheel commands can be told to avoid using the cache with the --no-cache-dir option. (eg: pip install --no-cache-dir <package>)
Since pip 10.0 (back in 2018!), a pip config command was added, which can be used to configure pip to always ignore the cache -- pip config set global.cache-dir false configures pip to not use the cache "globally" (i.e. in all commands).
Since pip 20.1, pip has a pip cache command to manage the contents of pip's cache.
pip cache purge removes all the wheel files in the cache.
pip cache remove matplotlib selectively removes files related to a matplotlib from the cache.
In summary, pip provides a lot of ways to tweak how it uses the cache:
pip install --no-cache-dir <package>: install a package without using the cache, for just this run.
pip config set global.cache-dir false: configure pip to not use the cache "globally" (in all commands)
pip cache remove matplotlib: removes all wheel files related to matplotlib from pip's cache.
pip cache purge: to clear all files from pip's cache.