I have created a single add-in, which targets 2003, 2007 & 2010 versions of Outlook.
I did this with VS2008, VSTOSE and PIA2003, along with Outlook 2003 on my dev machine. I'm not how this will scale up to office/document-level add-ins, but it may work.
You will, however, need to detect the version of Outlook/Office on each client and install the appropriate PIA.
This can be done using Component Checker. In each Bootstrapper package you can then check to see, which version exists and only install that package when applicable. For example in the product.xml for the bootstrapper package you would have:
<?xml version="1.0" encoding="utf-8"?>
<InstallChecks>
<ExternalCheck Property="Office2003Exists" PackageFile="ComponentCheck.exe" Arguments="{3EC1EAE0-A256-411D-B00B-016CA8376078}"/>
<ExternalCheck Property="Office2003PIAExists" PackageFile="ComponentCheck.exe" Arguments="{14D3E42A-A318-4D77-9895-A7EE585EFC3B}"/>
</InstallChecks>
<Commands Reboot="Defer">
<Command PackageFile="o2003pia.msi" Arguments="" EstimatedInstalledBytes="30000000" EstimatedInstallSeconds="60">
<InstallConditions>
<BypassIf Property="Office2003Exists" Compare="ValueNotEqualTo" Value="0" />
<BypassIf Property="Office2003PIAExists" Compare="ValueEqualTo" Value="0" />
<FailIf Property="AdminUser" Compare="ValueEqualTo" Value="false" String="AdminRequired"/>
</InstallConditions>
<ExitCodes>
<ExitCode Value="0" Result="Success"/>
<ExitCode Value="1641" Result="SuccessReboot"/>
<ExitCode Value="3010" Result="SuccessReboot"/>
<DefaultExitCode Result="Fail" FormatMessageFromSystem="true" String="GeneralFailure" />
</ExitCodes>
</Command>
</Commands>
The product ids for 2007 are:
Outlook : 0638C49D-BB8B-4CD1-B191-050E8F325736
PIA: ED569DB3-58C4-4463-971F-4AAABB6440BD
The product ids for 2010 are:
Outlook : CFF13DD8-6EF2-49EB-B265-E3BFC6501C1D
PIA: 1D844339-3DAE-413E-BC13-62D6A52816B2
This does mean you will have to include the pre-requisites into your installer package, rather than allowing for downloads, which will obviously increase download sizes.
You will also only be able to use methods etc that are in 2003. Also, any toolbars you create will be basic, as you won't have full control of the Ribbon. You can still add buttons etc as you would in 2003 & 2007. They will appear in their own ribbon group in 2010.
One word of advice though, in my solution, I created a separate assembly for anything that didn't interact with Outlook. That way, should requirements change in the future, I can easily split the add-ins to target specific versions, without affecting the main core functionality of the add-in.