How to track eyes using Kinect SDK?

9
votes

The requirement is to define a rectangle around each eye in 3D space. There should be a way to track eyes using the Microsoft Kinect SDK. According to this

The Face Tracking SDK uses the Kinect coordinate system to output its 3D tracking results. The origin is located at the camera’s optical center (sensor), Z axis is pointing towards a user, Y axis is pointing up. The measurement units are meters for translation and degrees for rotation angles.

Adding

...
Debug3DShape("OuterCornerOfRightEye", faceTrackFrame.Get3DShape()
    [FeaturePoint.OuterCornerOfRightEye]);
Debug3DShape("InnerCornerRightEye", faceTrackFrame.Get3DShape()
    [FeaturePoint.InnerCornerRightEye]);
Debug3DShape("InnerCornerLeftEye", faceTrackFrame.Get3DShape()
    [FeaturePoint.InnerCornerLeftEye]);
Debug3DShape("OuterCornerOfLeftEye", faceTrackFrame.Get3DShape()
    [FeaturePoint.OuterCornerOfLeftEye]);
...
private void Debug3DShape(string s, Vector3DF v)
{
    Debug.WriteLine(s + " X " + v.X + " Y " + v.Y + " Z " + v.Z);
}

to CreateResult() in Microsoft.Kinect.Toolkit.FaceTracking prints

OuterCornerOfRightEye X -0.05728126 Y 0.04850625 Z -0.1144406
InnerCornerRightEye X -0.01584376 Y 0.04850625 Z -0.1279687
InnerCornerLeftEye X 0.01584374 Y 0.04850625 Z -0.1279687
OuterCornerOfLeftEye X 0.05728124 Y 0.04850625 Z -0.1144406

when the SDK starts tracking a face. I should be able to use these coordinates to paint a box around each eye but the Z coordinate should probably be closer to 1.0, not -0.1.. or -0.2.. (based on my setup) so I don't trust the numbers. Is the XYZ supposed to be the location of the FeaturePoint in 3D space relative to the sensor? Do I misunderstand the Kinect coordinate system? Am I using the Kinect SDK incorrectly? Does it matter that I am using an Xbox 360 Kinect Sensor? (Microsoft does not guarantee full compatibility for Kinect for Windows applications and the Xbox 360 Kinect Sensor)

Edit: Adding this

if (trackSucceeded) {
    ...
    if (headPointsObj != null) 
        for (int i = 0; i < 2; i++) 
            DebugHeadPoint(i, headPointsObj.Points);
}
private void DebugHeadPoint(int i, Vector3DF[] points) {
    if (points == null) throw new ArgumentNullException("points");
    Debug.WriteLine("HeadPoint[" + i + "] X " + points[i].X + 
        " Y " + points[i].Y + 
        " Z " + points[i].Z);
}

to FaceTracker.cs::Track() prints this

HeadPoint[0] X 0.01227657 Y -0.2290326 Z 1.319978
HeadPoint[1] X 0.00613283 Y -0.02143053 Z 1.280334
HeadPoint[0] X 0.003939687 Y -0.2297621 Z 1.319813
HeadPoint[1] X -0.003732742 Y -0.02388078 Z 1.277723
...

These numbers seem correct based on the setup. The FeaturePoints print only once but the HeadPoints print continuously while trackSucceeded. Are FeaturePoint values relative to the HeadPoint?

1
c#kinectkinect-sdkeye-tracking
Outside of face tracking, XYZ is measured in millimeters from the Kinect origin. I would guess the values are the same, though they may be the difference from a facial point. I've not delt directly with the face tracker before to be sure. – Nicholas Pappas
@EvilClosetMonkey: According to this The measurement units are meters. – jacknad
Indeed, the seem to be in meters for the face tracker. The regular depth coordinate system reports in millimeters and I hadn't tracked down the link you posted above. How far away from the Kinect are you sitting? Grab a ruler! :) – Nicholas Pappas
I notice you got an answer to this on MSDN. It seems like the answer should be crossposted here for reference, but I'd feel odd doing it myself, as I didn't answer it. Perhaps the OP should? Not sure of the etiquette on this. – Mike Precup
@MikePrecup: I made a suggestion to the OP at MSDN to post a link here. – jacknad

1 Answers