I'm not sure what exactly the misunderstanding is. I'm noticing three strange points:
let aux ys = function ([], ys) -> declares and immediately re-declares, and thereby shadows, the identifier ys. Note that aux is a function with two curried arguments, of which the second is a 2-tuple. I doubt this is what you intended.
The aux function is indented in a rather unusual way; normally, it should get another four-space indentation. The compiler may not complain about this, and just exit the scope after the pattern match, but it adds to the confusion about what the failing line is supposed to do.
ys is undefined where it is last used. (Could this be related to the confusing indentation?)
Here are two ways to write it:
Not tail recursive
let rec inner xs ys =
match xs, ys with
| x::xt, y::yt -> x * y + inner xt yt
| _ -> 0
In this version, the curried arguments are turned into a tuple and used in a match expression. (Due to the final additions, this may cause a stack overflow for large input lists.)
Tail recursive, with auxiliary function
let rec private aux acc = function
| x::xs, y::ys -> aux (acc + x * y) (xs, ys)
| _ -> acc
let inner xs ys = aux 0 (xs, ys)
Here, the auxiliary function has two curried arguments, of which the first is an accumulator, and the second a tuple holding the two lists. inner becomes a wrapper function that both strips the accumulator – by initializing it with zero – and turns the tupled arguments into curried arguments, as was the requirement. (Since the recursive call's value is the functions return value, tail recursive compilation is supported for this function.)