Stencil Options between DirectX and OpenGL

Question

I'm implementing an emulation layer. Everything otherwise works, except the stencil stuff (my shadows overlap).

I'm just wondering if I'm doing things logically - if I'm making the proper conclusions/assumptions:

case D3DRS_STENCILENABLE:
    glAble(GL_STENCIL_TEST, value);
    break;
case D3DRS_STENCILFAIL:
    sStencilFail = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILZFAIL:
    sStencilPassDepthFail = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILPASS:
    sStencilPassDepthPass = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILFUNC:
    sStencilFunc = glFromD3DCMPFUNC((D3DCMPFUNC)value);
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILREF:
    sStencilRef = value;
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILMASK:
    sStencilValueMask = value;
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILWRITEMASK:
    AssertGL(glStencilMask(value));
    break;

The following are used above. glAble() simply is a wrapper for glEnable/glDisable.

static GLenum glFromD3DCMPFUNC(D3DCMPFUNC value) {
    return(GL_NEVER + value - 1);
}

static GLenum glFromD3DSTENCILOP(D3DSTENCILOP value) {
    switch (value) {
    case D3DSTENCILOP_KEEP: return(GL_KEEP);
    case D3DSTENCILOP_ZERO: return(GL_ZERO);
    case D3DSTENCILOP_REPLACE: return(GL_REPLACE);
    case D3DSTENCILOP_INVERT: return(GL_INVERT);
    case D3DSTENCILOP_INCRSAT:
    case D3DSTENCILOP_INCR:
        return(GL_INCR);
    case D3DSTENCILOP_DECRSAT:
    case D3DSTENCILOP_DECR:
        return(GL_DECR);
    default: Assert(!"Unsupported!"); return(0);
    }
}

Which versions of D3D and GL are you targeting? You may have trouble implementing D3D9 stenciling in versions of GL older than 2.0, which only expose two-sided stencil operations through extensions. I would expect to see something in your emulator that handles D3DRS_Two_Sided_StencilMODE and then dispatches things like D3DRS_CCW_STENCILFAIL to glStencilFuncSeparate (...) appropriately. It adds quite a bit of complication, but there are common use-cases for handling the front/back stencil test differently. — Andon M. Coleman

starmole starmole · Accepted Answer · 2013-10-15T04:17:38

There are a couple of things that could go wrong.

You do the same for incr and incr_sat. Is the test code using it? Gl needs an extension to saturate. It's not the same thing.
With old d3d code double check that the stencil ref is 0..255. There might be a difference how a parameter >255 or <0 is handled (clamp vs. and)
Double check that d3d is not doing two sided. And remember that the culling is flipped.
Double check the stencil clear code also!

The good news is, that this should work with basic table translation. There is no fundamental API issue (I did it before). It just needs a lot of testing because the parameter space is large.

Stencil Options between DirectX and OpenGL

2 Answers