Your question is a bit vague, and you should explain what "uninstall doesn't work" means, and why you think the service is anything to do with the app being open because the ServiceControl will stop the service.
Anyway, later versions of Windows Installer try to uninstall and leave the app running. It makes a private copy of the running app so that it can finish the uninstall, no reboot, and keep the app running. Any cleanup is non-critical and can be done at next reboot.
Without knowing what "doesn't work" means it's hard to say more, but if you need to shut it down look at the WiX util CloseApplication stuff.
The error message "There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected" almost certainly means that you are running a custom action that has failed, that's the first thing to check.