Display Color and Depth using OpenNI and OpenCV

Question

I've been posting quite alot recently about accessing depth images from a Kinect camera using OpenNI and OpenCV.

Following some tutorials and advice from some other posts I have been able to write this script for showing the 2D color stream of the camera (if the second part is commented out), and the depth stream, which is not yet working:

#include <opencv2/opencv.hpp>
#include <opencv2\highgui\highgui.hpp>
#include <opencv2\imgproc\imgproc.hpp>
#include <OpenNI.h>

int main()
{
    openni::Device device;

    openni::VideoStream  color;
    openni::VideoStream depth;

    openni::VideoFrameRef depthFrame;
    openni::VideoFrameRef colorFrame;

    openni::Status rc = openni::STATUS_OK;

    rc = openni::OpenNI::initialize();
    rc = device.open(openni::ANY_DEVICE);

    rc = color.create(device, openni::SENSOR_COLOR);
    rc = color.start();
    rc = depth.create(device, openni::SENSOR_DEPTH);
    rc = depth.start();

    cv::Mat framecolor;
    cv::Mat framedepth;

    while (true)
    {
        color.readFrame(&colorFrame);
        const openni::RGB888Pixel* imageBuffer = (const openni::RGB888Pixel*)colorFrame.getData();

        framecolor.create(colorFrame.getHeight(), colorFrame.getWidth(), CV_8UC3);
        memcpy(framecolor.data, imageBuffer, 3 * colorFrame.getHeight()*colorFrame.getWidth() * sizeof(uint8_t));

        cv::cvtColor(framecolor, framecolor, CV_BGR2RGB); //this will put colors right
        cv::imshow("framecolor", framecolor);
        ////////////////////////////Second Part/////////////////////////////////////
        depthFrame.getVideoMode();
        const openni::DepthPixel* imageBuffer2 = (const openni::DepthPixel*)depthFrame.getData();

        framedepth.create(depthFrame.getHeight(), depthFrame.getWidth(), CV_16U);
        memcpy(framecolor.data, imageBuffer2, 3 * depthFrame.getHeight()*depthFrame.getWidth() * sizeof(uint16_t));
        framedepth.convertTo(framedepth, CV_8U);
        cv::imshow("framedepth", framedepth);

        if (cvWaitKey(30)>=0)
        {
            break;
        }
    }
    cv::destroyAllWindows();
    return 0;
}

The error im getting takes me to the openNI.g file, with the following:

I'm guessing that it has something to do with the file types or pixel formats. there is however alot of option and i'm not even certain that this method of doing it will even work.

Can anyone help with the required filetypes or procedures?

shawndfernandes shawndfernandes · Accepted Answer · 2018-05-25T04:39:20

there is an error in the code,

framedepth.create(depthFrame.getHeight(), depthFrame.getWidth(), CV_16U);
memcpy(**framecolor.data**, imageBuffer2, 3 * depthFrame.getHeight()*depthFrame.getWidth() * sizeof(uint16_t));
framedepth.convertTo(framedepth, CV_8U)

should be

framedepth.create(depthFrame.getHeight(), depthFrame.getWidth(), CV_16U);
memcpy(**framedepth.data**, imageBuffer2, 3 * depthFrame.getHeight()*depthFrame.getWidth() * sizeof(uint16_t));
framedepth.convertTo(framedepth, CV_8U)

Display Color and Depth using OpenNI and OpenCV

1 Answers