If i understand your problem correctly you want certain fields to appear/disappear according to the choice selected in the option set.
The way I see this working is by implementing a Javascript on the onChange event of the option set. you can do this from the 'Form Properties' when changing the layout of the form.
Below is a link showing some examples of possible useful commands that might help you:
http://www.powerobjects.com/blog/2011/01/14/crm-2011-useful-javascript-tidbits/
You can put your custom fields in a custom tab and then hide/disable it according to the value in the option set.
If you have any further problems feel free to ask :)
I hope this helps :)
Gdluck