The best way to do this is by use of an HttpModule. This enables the use of your custom master page for all application pages (i.e. pages in the LAYOUTS folder). It can be deployed using a feature and can be activated per web application (seeing as the httpmodule needs to be registered in the web.config it is web app scoped.)
By making it web app scoped, your end users will have a uniform user experience, instead of that single page looking like the front end of the site while all the other (Out of the box) application pages are still using the default sharepoint application.master.
For a code example and a more in depth explanation, look here.
P.S. You are getting errors using the code above because of missing content placeholders. You need to create a copy of your custom master page. Although styling can be the same, application pages use more/other ContentPlaceHolders than a front end master page.
Just copy your custom master page, rename it from CustomMaster.master to, say, CustomMasterEdit.master and use that for application page styling, SharePoint will throw an error telling you which placeholders are missing, keep adding the needed placeholders till the page works (there are 2 or 3 extra placeholders needed i believe).
P.P.S. To make sharepoint display errors that make sense, go the web.config
and look for the <SharePoint>
tag and change the callstack
attribute from false
to true
. Then, look for the customErrors
tag and set the mode
attribute to "off"
. To completely enable debugging, you can also enable ASP.NET tracing. Of course you should NOT do this in your production environment....
More info on modifying the web.config to make sharepoint display real error message can be found here.
and