This was asked months ago, but I am answering for future reference for anyone else who encounters this problem.
I was just bitten by this issue. All advice listed in this post, and in other posts on this forum (not to mention posts on other forums) does not work, at least for some of us. I finally figured out the real issue.
vim on cygwin, for whatever reason (at least this was the case for me) does not use the .vimrc you put in your directory. Let's say you copy the example one to your working directory, or copy some .vimrc from online. Or maybe you create a new one from scratch, and put all the settings the good people here and elsewhere recommend (set backspace = blahblah, set nocompatible, set this, set that). It doesn't work. Why? Because for whatever reason (at least in my case) vim isn't looking at the .vimrc you just created.
The solution is to FORCE vim to use a particular .vimrc, by passing in -u on the command line like so:
vim -u [/INSERT/PATH/TO/.vimrc]
For the love of all that is holy, DO NOT type square brackets or the words "/INSERT/PATH/TO/.vimrc" verbatim. Use your brain please.
Anyway, this solved my problems and I was able to use the default example .vimrc and get proper delete and backspace behavior while in insert mode, not to mention other goodies.
You might want to alias the vim command in your .bashrc like this:
alias vim='vim -u [/INSERT/PATH/TO/.vimrc]'