I don't believe the Unused timeout has anything to do with the Connection timeout. If the Unused timeout is too low, the factory has to keep closing and opening connections, but that only applies to connections in the Free pool, not the Active pool. Nevertheless, you want to avoid repeated opening/closing of connections as it will impact performance.
Unused Timeout
The Connection pool property Unused timeout defines how long a JMS connection will stay in the Free Pool before it is disconnected. This property has the default value of 1800 seconds (30 minutes). Basically, if a connection sits dormant in the Free pool for longer than 1800 seconds, it will be disconnected.
Connection Timeout
The time the application waits for a connection from the Free pool if the number of connections created from this factory already is equal to the factory's Maximum connections property. If a connection is put back in the free pool within this 3-minute period, the Connection Manager immediately takes it out of the pool again and passes it to the waiting application. However, if the timeout period elapses, a ConnectionWaitTimeoutException is thrown.
So the Connection timeout is basically how long your application waits for the next available connection, assuming the factory can't create new connections because it's maxed out. If you find yourself hitting this ceiling, increase the factory's Maximum connections property.