The compatibility modes in IE8 and IE9 are intended to allow you to do exactly what you're asking -- to test your site for all IE versions while only having a single version of IE on your machine.
It's a great idea in principle. Unfortunately, microsoft messed it up. While the compatibility modes do indeed replicate IE7 and IE8, they also have their own bugs and issues which do not occur either in the parent browser nor in the browser they are attempting to emulate.
In effect, the compatibility modes are actually another set of different browsers. You can't rely on them to accurately show you what your site looks like in IE7, but at the same time you do still need to test with them in case you have users who are running in compatibility mode for whatever reason.
You also asked about quirks mode and standards mode. Please note that "quirks mode" has nothing whatever to do with compatibility mode.
Quirks mode is triggered in IE by a site not having a <!DOCTYPE> declaration at the top of the page. This was introduced when standards mode was brought out with IE6, to allow sites to continue working in older versions of IE. Quirks mode is therefore effectively an IE5 compatibility mode. There is absolutely no reason to be using quirks mode in any site today, so you should always specify a doctype (preferably the HTML5 doctype). If your site looks good in quirks mode, and not in standards mode, then you need to fix your site.
The HTML5 doctype <!DOCTYPE html> will trigger standards mode in all versions of IE, from IE6 and up. In fact, this doctype was deliberately chosen by the HTML5 standards team specifically because it was the shortest possible doctype that would work in all existing browsers.
About the HTML5Shim: I had some trouble getting the basic shim to work. I never tracked down the problem, but I resolved it by using Modernizr instead. This has the same functionality as the basic shim, plus a bunch of other useful features for browser compatibility checking.
Your impression of what the shim does is slightly wrong: it doesn't replace elements with divs. What it actually does is create dummy versions of each of the new HTML5 elements, using Javascript. This exploits a quirk in IE whereby it recognises an element as valid HTML if an instance of it has been created via Javascript. (It's a bizarre quirk, but it's proved useful for the hackers to force IE to accept modern standards.
Hope that helps.
