There are few information about the internal structure of MPI allreduce operation and its relation to MPI barrier, and this few information is not consistent. So, I am trying to clarify the following two questions:
No MPI collectives have barrier semantics except MPI_Barrier, although some other operations have barrier semantics due to data dependencies. MPI_All{reduce,gather,toall} have, at least for non-zero counts, such data dependencies and thus will impart a barrier.
Thus, no, you should never add a barrier to an allreduce except if you want to barrier when the count is zero.
I know of performance reasons to add barriers before collectives on some supercomputers but this is really an implementation problem.
Synchronization and blocking are not the same thing. Non-blocking collectives synchronize when completed. See the MPI standard for more detailed definitions of these terms.
Does this help? If not, I'll add to it until it does.
