Kinect transform from Skeleton data to Depth data and back

Question

I am experimenting with the kinect API, and what i am trying (and failing) to achieve is the following:

starting off i get the skeleton data from the kinect and alculate the distance of the right hand of the user from the kinect

mRightHandPosition = skeletonFrame.SkeletonData[i].SkeletonPositions[NUI_SKELETON_POSITION_HAND_RIGHT];    
distance = sqrt(pow(mRightHandPosition.x, 2) + pow(mRightHandPosition.y, 2) + pow(mRightHandPosition.z, 2));

I transform the skeleton data of the right hand to depth data, in order to get the position of the hand in the (depth/color) image.

FLOAT curRightX = 0, curRightY = 0;
Vector4 pixelInSkeletonSpace;
NuiTransformSkeletonToDepthImage(mRightHandPosition, &curRightX, &curRightY, cDepthResolution);

having obtained the pixel position of the hand, i want to transform back that pixel into skeleton data and calculate again the distance of the object in that pixel (the hand) from the kinect. I would assume that doing that should give me approximately the same distance as before (with some small error of course), but it does not. Here is what i do:

//the position of the depth pixel in the mLockedRect.pBits array 
//i have set the depth sensor resolution to 320x240
int pixelPosition = 2 * ((int)curRightX + (int)curRightY * 320);
USHORT p;
//convert the two consecutive bytes to USHORT
p = (((unsigned short)mLockedRect.pBits[pixelPosition]) << 8) | mLockedRect.pBits[pixelPosition + 1];
//get the pixel in skeleton space
pixelInSkeletonSpace = NuiTransformDepthImageToSkeleton(LONG(curRightX), LONG(curRightY), p, cDepthResolution);
//calculate again the distance (which turns out completely wrong)
distance = sqrt(pow(pixelInSkeletonSpace.x, 2) + pow(pixelInSkeletonSpace.y, 2) + pow(pixelInSkeletonSpace.z, 2));

am i missing something obvious? thanks in advance

tk66 tk66 · Accepted Answer · 2014-12-21T02:06:39

After a lot of searching i found out what was wrong.. Here is the solution for anyone else who is trying to do something similar

first in order to save the depth data, the best way (i found)) was the following

in the processDepth() function:

bghr = m_pBackgroundRemovalStream->ProcessDepth(m_depthWidth * m_depthHeight * cBytesPerPixel, LockedRect.pBits, depthTimeStamp);
const NUI_DEPTH_IMAGE_PIXEL* pDepth = reinterpret_cast<const NUI_DEPTH_IMAGE_PIXEL*>(LockedRect.pBits);
memcpy(mLockedBits, pDepth, m_depthWidth * m_depthHeight * sizeof(NUI_DEPTH_IMAGE_PIXEL));

in the ComposeImage() function (or any function you want to use the depth data):

//transform skeleton data point to depth data
NuiTransformSkeletonToDepthImage(mRightHandPosition, &curRightX, &curRightY, cDepthResolution);

//calculate position of pixel in array
int pixelPosition = (int)curRightX + ((int)curRightY * m_depthWidth);

//get the depth value of the pixel
const USHORT depth = mLockedBits[pixelPosition].depth;

//create a new point in skeleton space using the data we got from the previous transformation
pixelInSkeletonSpace = NuiTransformDepthImageToSkeleton(LONG(curRightX), LONG(curRightY), depth << 3, cDepthResolution);

//calculate estimated distance of right hand from the kinect sensor using our recreated data
FLOAT estimated_distance = sqrt(pow(pixelInSkeletonSpace.x, 2) + pow(pixelInSkeletonSpace.y, 2) + pow(pixelInSkeletonSpace.z, 2));

//calculate the distance of the right hand from the kinect sensor using the skeleton data that we got straight from the sensor
FLOAT actual_distance = sqrt(pow(mRightHandPosition.x, 2) + pow(mRightHandPosition.y, 2) + pow(mRightHandPosition.z, 2));

now the estimated_distance and the actual_distance should have approximately the same values with some small variance.

Kinect transform from Skeleton data to Depth data and back

1 Answers