When you hit the F5 button on the keyboard the add-in is run under the debugger with non-optimized code. I'd suggest testing the RELEASE configuration of your add-in instead. In that case you will get an optimized assembly without debug symbols defined, so it should be run a lot faster.
The New in Outlook for developers page states the following:
Extending the add-in resiliency pillar of Outlook 2010, Outlook 2013 monitors add-in performance metrics such as add-in startup, shutdown, folder switch, item open, and invoke frequency. Outlook records the elapsed time in milliseconds for each performance monitoring metric.
For example, the startup metric measures the time required by each connected add-in during Outlook startup. Outlook then computes the median startup time over 5 successive iterations. If the median startup time exceeds 1000 milliseconds (1 second), then Outlook disables the add-in and displays a notification to the user that an add-in has been disabled. The user has the option of always enabling the add-in, in which case Outlook will not disable the add-in even if the add-in exceeds the 1000 millisecond performance threshold