Okay, this is killing me. I've written a WordPress plugin that adds a custom button to the TinyMCE editor and it works great, except that the options pop-up window triggered by the button loads differently than similar popup windows that are native to WP...
Specifically, when you click the Add Media button above the TinyMCE toolbar in WordPress, the Add Media window opens up in a nice thickbox modal. It also loads in an iframe, because the button HTML that gets output is wrapped in an anchor tag with a URL href value:
<a href="http://.../wp-admin/media-upload.php?post_id=916&TB_iframe=1&width=640&height=354" class="thickbox add_media" id="content-add_media" title="Add Media" onclick="return false;">
But when my button's HTML gets output, the anchor tag has an href value of 'javascript:;':
<a role="button" id="content_companyinfo" href="javascript:;" class="mceButton mceButtonEnabled mce_companyinfo" onmousedown="return false;" onclick="return false;" aria-labelledby="content_companyinfo_voice" title="Insert company info fields" tabindex="-1">
Here is my TinyMCE editor plugin code: http://pastebin.com/kuQJRHJR
And here is the WP core TinyMCE editor plugin for the Add Media button: http://core.trac.wordpress.org/browser/trunk/wp-includes/js/tinymce/plugins/media/editor_plugin_src.js
The TinyMCE addButton function doesn't allow you to set an href value, only a cmd value that links to a custom command function, which you create using addCommand.
I tried just adding the class 'thickbox' to my button in the settings of the addButton function, but that had a weird effect of creating a thickbox modal behind my normal pop-up window.
Anyone know how to do this the right way? Thanks in advance!