Christian, You said yourself, Package and Theme is set in "Default Config" scope. When you place the file in /default/custom/layout/ you are placing the file in a different template folder. You can do one of two (three) things:
Change Design Settings
Under System > Configuration > Design > Themes, set Templates, Skin, and Layout to "custom".
Refresh your cache and it should start using the folders /design/frontend/default/custom/...
Use Default/Default Theme Package
You can just edit the Default theme. You can copy the files over from "Base/Default" to "Default/Default" and set it to not overwrite any files, then move your files from "Default/Custom" and overwrite over the default folder.
Note: This gets the job done, but isn't generally recommended. (Keep Reading ;D)
Change Design Settings and Use local.xml
Change your configuration as stated above. The most accepted way of changing the layout in Magento nowadays is to use local.xml. This would be located in your Default/Custom/Layout/ folder. This one file will be where all your layout updates will be done, and you won't have to touch any core files in the process.
Some things you're used to doing will have to change, since you won't be editing the base files directly. The local.xml gets loaded last, so any adjustments made here shouldn't be overwritten. To accomplish what you're aiming to do, your local.xml might start off something like this:
local.xml
<?xml version="1.0"?>
<layout version="0.1.0">
<checkout_onepage_success>
<reference name="root">
<action method="setTemplate">
<template>page/1column.phtml</template>
</action>
</reference>
</checkout_onepage_success>
<checkout_onepage_failure>
<reference name="root">
<action method="setTemplate">
<template>page/1column.phtml</template>
</action>
</reference>
</checkout_onepage_failure>
</layout>
You will have to adjust the way you remove things, it's no longer as simple as commenting out, deleting, or moving a line though.
Remove a Block Completely
To remove a block, get it's block name (or as="") and insert the below code in the appropriate reference.
<remove name="left.permanent.callout" />
Move a Block Elsewhere
Moving blocks around is a two parter, first you must unsetChild in the containing reference and insert the block in its new location. For example:
<reference name="left">
<action method="insert">
<blockName>right.poll</blockName>
<siblingName>left.newsletter<siblingName>
<after>0</after>
</action>
</reference>
<reference name="right">
<action method="unsetChild">
<name>right.poll</name>
</action>
</reference>
Note: Here, the 0 typically only applies to blocks where their phtml file contains a echo $this->getChildHtml(''). This means it is loading all referenced child blocks as listed in the xml.
Also note, the files that use the same function using the block name (e.g. getChildHtml('top_links')) will typically require you to clone that template file into your design (Default/Custom/Template/) and manually add echo $this->getChildHtml('your_block_name') where you wish your block to appear (after placing it in your local.xml).
That should get you started, there's plenty of good articles online and stack overflow is a good place for info as well.
checkout.xml
copied to your custom theme then editing/base/default/layout/checkout.xml
will have no effect. Try editing the copy of checkout.xml that you copied to your own custom theme. – seanbreeden/default/default/layout/checkout.xml
– seanbreeden