Sam has described the problem, but as for a general solution, you basically need to teach Emacs to distinguish the ESC received in response to you hitting the "escape key" from the ESC received as part of an "escape sequence". Of course, in general it can't be done, but in practice, you can check the timing: if ESC is followed by a bit of idle time, then it's probably an "escape key", and otherwise it's probably part of an "escape sequence".
This trick is used in VI emulators such as Viper or Evil:
(defvar viper-fast-keyseq-timeout 200)
(defun viper--tty-ESC-filter (map)
(if (and (equal (this-single-command-keys) [?\e])
(sit-for (/ viper-fast-keyseq-timeout 1000.0)))
[escape] map))
(defun viper--lookup-key (map key)
(catch 'found
(map-keymap (lambda (k b) (if (equal key k) (throw 'found b))) map)))
(defun viper-catch-tty-ESC ()
"Setup key mappings of current terminal to turn a tty's ESC into `escape'."
(when (memq (terminal-live-p (frame-terminal)) '(t pc))
(let ((esc-binding (viper--lookup-key input-decode-map ?\e)))
(define-key input-decode-map
[?\e] `(menu-item "" ,esc-binding :filter viper--tty-ESC-filter)))))
If you call viper-catch-tty-ESC, it will setup the decoding such that hitting the escape key should now generate an escape event (instead of an ESC event). This will automatically be mapped back to ESC if there is no binding for escape, thanks to a binding in function-key-map (this is used in GUI mode where the escape key indeed sends to escape event).
Note that this will not fix your problem: "ESC ESC up" will still insert "OA". The problem there is that Emacs's keyboard translation will still see "ESC ESC ESC O A" (tho the first two appeared in a round-about way going through escape and back). So to finally fix the problem, you additionally need to remove the "ESC ESC ESC" binding and replace it with a binding that will only be triggered with the new escape event:
(global-unset-key [?\e ?\e ?\e])
(global-set-key [?\e ?\e escape] 'keyboard-escape-quit)
Note: This is all tricky business. I'm intimately familiar with the corresponding code, yet my first two attempts while writing this answer failed because of some interaction I did not anticipate.