FBO errors when getting stencil buffer using OSG

Question

I'm trying to get the stencil index of a node using offline rendering in OSG. My main procedures are listed below:

1. deeply copy the node I want to offline render and get the stencil index;
2. create a offline render camera, and make the previous copied node as the camera's child, and then add this camera as child node to the original node I want to offline render;
3. create a DrawCallback which gets the stencil buffer and checks stencil indices of every pixel, and get the maximum stencil index.

Ok, the procedures seem a bit long. Now I can get the correct maximum stencil index, however, I get the warnings and errors:

"RenderStage::runCameraSetUp(), FBO setup failed, FBO status=0x8cdd" "Warning: detected OpenGL error' invalid operation' at end of SceneView::draw() ..."

The whole codes are a bit long and tedious. Some main code snippets are listed below:

void OfflineCallback::operator()(osg::RenderInfo &renderInfo)const
{
    osg::Image *image = new osg::Image;
    image->readPixels(0, 0, 512, 512, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE);
    unsigned char *dtBuf = (unsigned char *)(image->getDataPointer());
    int maxStencil = 0;
    for (int i = 0; i != image->s(); ++i){
        for (int j = 0; j != image->t(); ++j){
            unsigned char dt = dtBuf[image->s() * i + j];
            if(dt > maxStencil)
                maxStencil = dt;
        }
    }
}

Then I create the offline render camera, and attach the stencil buffer:

osg::Camera *camera = new osg::Camera;
camera->setClearMasks(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT|GL_STENCIL_BUFFER_BIT);
camera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
camera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
camera->attach(osg::Camera::STENCIL_BUFFER, GL_UNSIGNED_BYTE);

osg::Stencil *stencil = new osg::Stencil;
stencil->setFunction(osg::Stencil::GREATER, 1, 0xff);
stencil->setOperation(osg::Stencil::INCR, osg::Stencil::INCR, osg::Stencil::INCR);
osg::Depth *depth = new osg::Depth;
depth->setFunction(osg::Depth::ALWAYS);

osg::StateSet *ss = camera->getOrCreateStateSet();
ss->setAttributeAndModes(stencil, osg::StateAttribute::ON);
ss->setAttributeAndModes(depth, osg::StateAttribute::ON);
camera->setPostDrawCallback(new OfflineCallback);

To test the codes, I create a sphere as the main node, and then I get the maximum stencil index as 2 which is correct but with the errors and warnings mentioned at the beginning.

What am I missing? Which step is wrong? Any suggestion would be appreciated. Thanks!

Sincerely, Jimmy

Answer 1

This call looks suspicious:

camera->attach(osg::Camera::STENCIL_BUFFER, GL_UNSIGNED_BYTE);

I haven't used OSG, but based on the documentation, the second argument to this method is internalFormat, which corresponds directly to the OpenGL GLenum value for the internal format of the stencil attachment. GL_UNSIGNED_BYTE is not a valid value internal format for a stencil buffer. This needs to be GL_STENCIL_INDEX, or one of its sized variants. For example:

camera->attach(osg::Camera::STENCIL_BUFFER, GL_STENCIL_INDEX8);

Beyond this, 0x8cdd is the value of GL_FRAMEBUFFER_UNSUPPORTED. This means that (slightly paraphrased from the spec), "the combination of internal formats of the attached images violates an implementation-dependent set of restrictions."

OpenGL implementations can give this error for any combination of attachments that is not specified as required, as documented in section "9.4.3 Required Framebuffer Formats" in the OpenGL 4.5 spec.

If you still get this error after fixing the stencil format, I strongly suspect that your attempt fails because you are using separate buffers for depth and stencil. This case is specifically mentioned in section 9.4.3:

However, when both depth and stencil attachments are present, implementations are only required to support framebuffer objects where both attachments refer to the same image.

It is common for vendors to only support combined depth/stencil when both depth and stencil are required. Internally, there is typically a single 32 bits/pixel buffer with 24 bits of depth and 8 bits of stencil data.

So if you currently have something like this:

camera->attach(osg::Camera::DEPTH_BUFFER, GL_DEPTH_COMPONENT24);
...
camera->attach(osg::Camera::STENCIL_BUFFER, GL_STENCIL_INDEX8);

you should use a combined depth/stencil buffer if you want your code to work across most platforms:

camera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, GL_DEPTH24_STENCIL8);

Answer 2

OK, finally I get the correct stencil index and more importantly without any errors and warnings!

Reto's answer is still quite useful. Thank you again, Reto! His main point is if you want to get both the depth & stencil buffers at the same time, most vendors will only provide you the packed depth stencil buffer instead.

So in other words, if you attach the depth & stencil buffer to a FBO independently at the same time, you would largely fail to create the FBO. That's the actual reason why I get the errors and warnings at the first place.

The detailed reasons are: The OSG camera implicitly creates color & depth buffers while not creates the stencil buffer. That's fine for most cases. However, in my case, I explicitly want the stencil buffer. So at last, OSG camera creates all the three buffers for me. Since as previously mentioned, independent depth & stencil buffers would result in FBO creation failure, so I receive all the warnings and errors.

Solution:

image->allocateImage(512, 512, 1, GL_STENCIL_INDEX, GL_UNSIGNED_BYTE);
camera->attach(osg::Camera::STENCIL_BUFFER, image);
//explicitly tells camera not to implicitly create color & depth buffers
camera->setImplicitBufferAttachmentRenderMask(0);

And in the DrawCallback, you can read the stencil indices easily, just like reading depth buffer.

If anyone has other ideas or corrections, I am all ears!