Isn't it simple to have just left or right associativity for all the
same precedence operators?
Yes and it is the case in C. May be you assumed that prefix and postfix have the same precedence which is wrong. Postfix has a higher precedence than prefix!
Also there is another curious case to consider as to why certain operators have certain associativity. From Wiki,
For example, in C, the assignment a = b is an expression that returns
a value (namely, b converted to the type of a) with the side effect of
setting a to this value. An assignment can be performed in the middle
of an expression. (An expression can be made into a statement by
following it with a semicolon; i.e. a = b is an expression but a = b;
is a statement). The right-associativity of the = operator allows
expressions such as a = b = c to be interpreted as a = (b = c),
thereby setting both a and b to the value of c. The alternative (a =
b) = c does not make sense because a = b is not an lvalue.