Reals are not an equality type in SML, so (m1-m2) = 0 is a type error.
The reason for this is that the limited precision of floating-point representations can give unexpected results due to rounding errors (e.g. (1.0/7.7)*7.7 = 1.0 would return false). You can get around this by using the == operator from the Real library, i.e. Real.==(m1-m2,0) (or just Real.==(m1,m2)). But keep in mind that it can be unreliable.
The second problem is that, according to the return type, your function is supposed to return a value, not print it. All you need to do here is state the return value in the else clause, i.e. just replace print((x,y)) with (x,y).
And for what it's worth, I'd avoid using exceptions if you can; they kind of go against the idea of functional programming. Try returning a (real*real) option instead.
mis the gradient,bis the y-intercept. - Nick Barnes