How does one use clip() to perform alpha testing?

Question

This is an HLSL question, although I'm using XNA if you want to reference that framework in your answer.

In XNA 4.0 we no longer have access to DX9's AlphaTest functionality.

I want to:

1. Render a texture to the backbuffer, only drawing the opaque pixels of the texture.
2. Render a texture, whose texels are only drawn in places where no opaque pixels from step 1 were drawn.

How can I accomplish this? If I need to use clip() in HLSL, how to I check the stencilbuffer that was drawn to in step 1, from within my HLSL code?

So far I have done the following:

_sparkStencil = new DepthStencilState
{
    StencilEnable = true,
    StencilFunction = CompareFunction.GreaterEqual,
    ReferenceStencil = 254,
    DepthBufferEnable = true
};


DepthStencilState old = gd.DepthStencilState;
gd.DepthStencilState = _sparkStencil;

// Only opaque texels should be drawn.
DrawTexture1();

gd.DepthStencilState = old;

// Texels that were rendered from texture1 should
// prevent texels in texture 2 from appearing.
DrawTexture2();

Answer 1

Sounds like you want to only draw pixels that are within epsilon of full Alpha (1.0, 255) the first time, while not affecting pixels that are within epsilon of full Alpha the second.

I'm not a graphics expert and I'm operating on too little sleep, but you should be able to get there from here through an effect script file.

Answer 2

To write to the stencil buffer you must create a DepthStencilState that writes to the buffer, then draw any geometry that is to be drawn to the stencil buffer, then switch to a different DepthStencilState that uses the relevant CompareFunction.

If there is some limit on which alpha values are to be drawn to the stencil buffer, then use a shader in the first pass that calls the clip() intrinsic on floor(alpha - val) - 1 where val is a number in (0,1) that limits the alpha values drawn.

I have written a more detailed answer here:

Stencil testing in XNA 4