This question is a bit old. But I think the answer is that VS2010 SP1 has SQLCE 3.5 and SQLCE 3.5 x64 only, and not SQL Compact 4.0. The SQL products come together with Studio. There is still no SQLCE 4.0 today in 2012 included in distribution of Studio 2010 (I use Express edition).
This version packaging is not direct root cause, but just a clue, that different version will need extra effort.
The fix should involve registration of "Data Provider" in .NET Framework used by Visual Studio.
Look at "DbProviderFactories" inside c:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config. This section describes what data providers can be used by .NET and Studio.
If there is unexpected entries there, try to repair framework or register providers manually. To do so, some low level registration tools will be required. Say for asp.net providers there is aspnet_regsql.exe etc. But for just SQLCE itself there must be something similar.
<system.data>
<DbProviderFactories>
<add name="Microsoft SQL Server Compact Data Provider" invariant="System.Data.SqlServerCe.3.5" description=".NET Framework Data Provider for Microsoft SQL Server Compact" type="System.Data.SqlServerCe.SqlCeProviderFactory, System.Data.SqlServerCe, Version=3.5.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"/>
</DbProviderFactories>
</system.data>
The difference between broken Studio and working App is that both use the same Framework but have different providers in config. (Or app uses other technique to pick a DLL and open SDF)
Answer: Since SP1 it is officially possible to use SQLCE4.0 in Studio 2010 SP1 (Thanks to Nathan Fisher for link http://support.microsoft.com/kb/983509 )