A very coarse approximation is if your thread of execution needs to block while holding a resource (ie. mutex, sem), you shouldn't use a mutex. The problem with that approximation is that block means something different in a UI program and an interrupt handler; blocking is a relative notion.
Mutexes are bound to owners; Semaphores are not. The only agent that can release a mutex is the one that acquired it. Semaphores do not have this limitation. Because of this limitation, if a low priority agent blocks a high priority agent on a mutex, the supervisor (kernel, whatever..) can boost the priority of the owner until it relinquishes it. This can be applied transitively to solve non-cyclic priority inversion. You cannot do this with semaphores, since they lack the concept of an owner.
For example, thread 1 can acquire a semaphore, thread 2 can wait on it, and thread 3 can relinquish it. That might sound like chaos, but it might be the foundation of a more sophisticated system where thread 3 and 1 are communicating via some means, so thread 1 can hand ownership of a resource over to thread 3 directly. This can't be done with mutexes.
That said; I could like hack a pthread_mutex_transfer(mutex, pthread) into any existing implementation, thumbing my nose at the purists and theoreticians.