Either go with a JavaScript button that takes you to a Visualforce page or you'll need to have a Visualforce page that extends Case which immediately redirects you to a non-case Visualforce page. Since that's ugly and unnecessary, just make a JavaScript button on your Case object (name|setup|app setup|customize|cases|buttons and links|new then select OnClick JavaScript for Content Source).
Then in JavaScript:
parent.location.href ='/apex/YourNonCaseVisualforcePage';
EDIT: Check out this tutorial for tips on sending data back from a popup to Visualforce.