The short answer to "Why does create_mutable/2 not throw an exception when output unification fails?" is just: Because this was how it was done when the feature was added to SICStus Prolog, and no one has made a strong case for changing this.
One important "difference between the stream created by open/4 and the mutable term created by create_mutable/2" is that open/4 has side-effects that are not undone if the output-unification of the call to open/4 fails.
In this sense, create_mutable/2 is somewhat more like is/2 which also just quietly fails if the output argument is some non-numeric non-variable term, e.g. in x is 3+4. This seems to be the common, and traditional, way of handling output arguments in Prolog.
I agree that a non-variable as second argument is most likely a programming error. The next version of the SICStus IDE, SPIDER, will warn for this (as it already does for is/2).
None of this, nor the example in the question, seems directly related to the cited documentation "[...] the effect of unifying two mutables [...]".
open/4and the mutable term created bycreate_mutable/2... will reread the text you linked. - repeatuninstantiation_error, indeed. - false