I think the difference is because of the heap.
MSDN says: Processes that the debugger creates (also known as spawned processes) behave slightly differently than processes that the debugger does not create.
Instead of using the standard heap API, processes that the debugger creates use a special debug heap. You can force a spawned process to use the standard heap instead of the debug heap by using the _NO_DEBUG_HEAP environment variable or the -hd command-line option.
When using debug heap, it will check the heap integrity when allocation or free, so it will impact the performance.
Also, it disable the low-fragmentation heap when enable the debug heap. If you run your exe directly, the system uses the low-fragmentation heap (LFH) as needed to service memory allocation requests after Windows Vista, LFH can improve the application performance greatly if your application has a lot of allocation and free.
I encounter the same issue when I tried to improve the application performance under Window XP long time ago, while the LFH is disable by default on XP. I added the code to enable the LFH in my application, then I found out the code only worked when the application started from the explorer, not work when started with debugging in VS.