I came across this question looking for the same solution: The requirement is that customers in group X get 10% discount, but exclude products that already have discount, either via catalog price rules of special_price value.
There seems to be no clear cut solution, until now ;)
As noted in other answers, and comments, you can exclude products from the discount rule using a category.
This is a viable solution, but the issue is the need to maintain that list of products, which can be a very time-consuming, and error-prone task.
It can be automated!
Using the Dynamic Category Products extension (developed by myself), you can automate the list of products that has active discounts applied.
A quick howto.
In your store, create a new Root Category.
This is simply to group your 'data' categories outside the default store view, so they don't accidentally show up anywhere.
you then create sub-categories from this new root category. One of those will be a category that will list all the products on discount.
As an example, here is my Root Data Category, with some sub-categories. I use these all in shopping cart rules.

For this answer, we would be interested in the All products with discount applied
This category contains Dynamic Rules as follows:

The category will be dynamic, self updating, daily, and can be used to exclude products from existing Shopping Cart Discounts.
I hope this is of help to someone.