Kinect v2 - Synchronize depth and color frames

Question

I am currently looking for a stereoscopic camera for a project and the Kinect v2 seems to be a good option. However, since it's quite an investment to me, I need to be sure it meets my requirements, the main one being a good synchronization of the different sensors.

Apparently there is no hardware synchronization of the sensors, and I get many versions about the software part:

Some posts where people complain about lag between the 2 sensors, and many others asking for a way to synchronize the sensors. Both seem to have strange workarounds and no "official", common solution emerges from the answers.
Some posts about a MultiSourceFrame class, which is part of Kinect SDK 2.0. From what I understand, this class enables you to retrieve the frame of all the sensors (or less, you can choose which sensors you want to get the data from) at a given time. Thus, you should be able, for a given instant t, to get the output of the different sensors and make sure these outputs are synchronized.

So my question is, is this MultiSourceFrame class doing exactly what I mean it does? And if yes, why is it never proposed as a solution? It seems the posts of the 1st category are from 2013, so before the release of the SDK 2.0. However, MultiSourceFrame class is supposed to replace the AllFramesReady event of the previous versions of the SDK, and AllFramesReady wasn't suggested as a solution either.

Unfortunately the documentation doesn't provide much information about how it works, so I'm asking here in case someone would have already used it. I'm sorry if my question seems stupid, but I would like to be sure before purchasing such a camera.

Thank you for your answers! And feel free to ask for more details if needed :)

...but the larger problem with the Kinect v2 are the hardware requirements, in particular the USB 3.0 controller. You'll never know for sure if your's works unless you try it. — HenningJ
@HenningJ Thank you so much for your answer! Never found this issue before, so it's a real help here. Could you detail a bit more about this USB 3.0 issue? — Sonia Seddiki
According to the official requirements, you need an Intel or Renesas USB 3.0 controller. But even then it's hard to predict who well it will work, i.e. if it will work at all, or if it does, how many FPS you will be getting. The only way to know for sure is to try it. If it doesn't work with you built-in USB 3 controller, there are lots of USB 3 PCIe extension cards that should work. — HenningJ
@SoniaSeddiki we have developed an open-source software that acquires Kinect v2 data synchronously. Feel free to use it — 16per9

HenningJ HenningJ · Accepted Answer · 2016-11-16T15:28:20

There was a discussion about that in an libfreenect2 issue, where someone specifically mentioned a 6.25 millisecond lag between RGB and depth frame when using the MultiSourceFrameReader:

The RelativeTime of the ColorFrame seems to always lags 6.25 or 6.375 ms behind the RelativeTime of the DepthFrame, InfraredFrame, BodyFrame, BodyIndexFrame. Meanwhile, the RelativeTime always matches among DepthFrame, InfraredFrame, BodyFrame, and BodyIndexFrame.

In my own experiments, I got the same results. But that's only based on the frame's timestamps. These timestamps come from the Kinect v2 device directly, so it's unlikely, but still be possible that they are not 100% correct.

So, while there is a lag between depth and RGB frames, even when using MultiSourceFrameReader, it's most likely small enough so you can ignore it.

As for the usage of MultiSourceFrame/MultiSourceFrameReader, it's pretty simple once you got used to the Kinect v2 SDK:

m_pKinectSensor->OpenMultiSourceFrameReader(
    FrameSourceTypes::FrameSourceTypes_Depth | FrameSourceTypes::FrameSourceTypes_Color,
    &m_pMultiSourceFrameReader);

// get "synced" frame
IMultiSourceFrame* pMultiSourceFrame = NULL;
m_pMultiSourceFrameReader->AcquireLatestFrame(&pMultiSourceFrame);

// get depth frame
IDepthFrameReference* pDepthFrameReference = NULL;
pMultiSourceFrame->get_DepthFrameReference(&pDepthFrameReference);
IDepthFrame* pDepthFrame = NULL;
pDepthFrameReference->AcquireFrame(&pDepthFrame);

// get RGB frame
IColorFrameReference* pColorFrameReference = NULL;
pMultiSourceFrame->get_ColorFrameReference(&pColorFrameReference);
IColorFrame* pColorFrame = NULL;
pColorFrameReference->AcquireFrame(&pColorFrame);

// ... now use both frames

You can find more details in the CoordinateMapping Basic sample, once you installed the Kinect v2 SDK.

Kinect v2 - Synchronize depth and color frames

2 Answers