There are two ways to write a list (and a list pattern). If you use the :: operator you can have an arbitrary list at the tail of your pattern. If you use the [ ... ] notation, your pattern will only match a list of a specific length.
You are mixing these two pattern types, which is not what you want. (You can of course mix them when matching against an appropriate type.)
Your pattern should (most likely) look like this: Poly ((x, y) :: xs). Note the parentheses around the body of the pattern. The :: operator has lower precedence than the application of Poly. So you need the parentheses.
