Kinect sideways skeleton tracking

Question

Currently I am using the Microsoft Kinect for measuring angles between joints. Most measurements are working correctly. Whenever a person is sitting sideways (on a chair) the Kinect won't track the skeleton accurate. To illustrate my problem I've added 3 pictures of the Kinect depthview.

Seated sideways measurement with skeleton tracking

Seated sideways measurement without skeleton tracking

Sideways measurement with skeleton tracking

As you can see 2 out of 3 measurements work "correctly". Whenever I lift my leg, the Kinect stops skeleton tracking correctly. Does anyone have a solution to this problem, or is this just a limitation of the Kinect?

Thanks.

Update 1: The JointTrackingState-Enumeration on these tracked joints shown at screenshot 2 are marked as Inferred, however the depth view is tracking my full body.

Update 2: At screenshot 2 I'm trying to track my front leg, highlighted as green. I know the other leg isn't tracked but this does not matter, I guess.

Update 3: The following code selects a skeleton:

private Skeleton StickySkeleton(Skeleton[] skeletons)
{
    if (skeletons.Count<Skeleton>(skeleton => skeleton.TrackingId == _trackedSkeletonId) <= 0)
    {
        _trackedSkeletonId = -1;
        _skeleton = null;
    }

    if (_trackedSkeletonId == -1)
    {
        Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton => skeleton.TrackingState == SkeletonTrackingState.Tracked);

        if (foundSkeleton != null)
        {
            _trackedSkeletonId = foundSkeleton.TrackingId;
            return foundSkeleton;
        }
    }

    return _skeleton;
}

Whenever a skeleton is tracked that data will be used for drawing the joint-points and calculating the angle between joints.

Update 4: I've tested sitting on a 'block', far more simplified than a chair. Unfortunately the Kinect still acts the same.

Below 2 screenshots:

Sitting on a block 1

Sitting on a block 2

I have had many problems, when i sit on something or took something in hands. So i think that this is limitation of the kinect. — Likurg
@MarkvanVelthoven imo the problem is that Kinect has no estimation to the floor. On screen 1 and 3 youre touching the floor, kinect is using this to mesure everything. On screen 2 youre "in the air" does not know what to do. That is my imo — Fixus
So what about games like: Kinect Sports. They're jumping and therefor "in the air" and the Kinect still knows what to do. Any suggestions? — Velth
I've tried KinectExplorer sample containted inside the Microsoft Kinect SDK. The following screenshots show the behaviour of the Kinect using the KinectExplorer sample: Screenshot 1 and Screenshot2. As you can see, same result. See my main post for code, thanks! — Velth
This is a limitation in the Kinect SDK (Skeletal tracking) however you are free to place in your own skeletal tracking algorithm from the depth image. I am not going to place this as an answer but hope this excels how you think about the issue. This is how I would do it: — davidbates

Liam McInroy Liam McInroy · Accepted Answer · 2012-04-21T16:05:02

As Renaud Dumont stated, I would do something with JointTrackingState. Since your using knees, I used the variables leftknee and rightknee which are Joints to do it. Here's code, You might use JointType.FootRight and JointType.FootLeft and the Hip types, but I'll leave that up to you.

static Skeleton first = new Skeleton();

Joint leftknee = first.Joints[JointType.KneeLeft];
Joint rightknee = first.Joints[JointType.KneeRight];

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

Or alternately, if you just wanted one knee to be tracked at a time, or both, you could do this:

if ((leftknee.TrackingState == JointTrackingState.Inferred ||
                leftknee.TrackingState == JointTrackingState.Tracked) && 
                (rightknee.TrackingState == JointTrackingState.Tracked ||
                rightknee.TrackingState == JointTrackingState.Inferred))
            {

            }

            else if (leftknee.TrackingState == JointTrackingState.Inferred ||
                    leftknee.TrackingState == JointTrackingState.Tracked)
            {

            }

            else if (rightknee.TrackingState == JointTrackingState.Inferred ||
                    rightknee.TrackingState == JointTrackingState.Tracked)
            {

            }

FYI the reason the Skeleton is first is static is because then it can be used in making the joints

 static Skeleton first;

Opposed to

 Skeleton first;

Edit 1

extremelyclassSkeletalTrackingStateInferredenumenumimpossibleinherit.dll

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Kinect;

namespace IsInferred
{
public abstract class SkeletonInferred : Skeleton
{
    public bool inferred;
    static Skeleton first1 = new Skeleton();
    Joint handright;
    Joint handleft;
    Joint footright;
    Joint footleft;
    Joint ankleleft;
    Joint ankleright;
    Joint elbowleft;
    Joint elbowright;
    Joint head;
    Joint hipcenter;
    Joint hipleft;
    Joint hipright;
    Joint shouldercenter;
    Joint shoulderleft;
    Joint shoulderright;
    Joint kneeleft;
    Joint kneeright;
    Joint spine;
    Joint wristleft;
    Joint wristright;

    public SkeletonInferred(bool inferred)
    {

    }

    public enum Inferred
    {
        NotTracked = SkeletonTrackingState.NotTracked,
        PositionOnly = SkeletonTrackingState.PositionOnly,
        Tracked = SkeletonTrackingState.Tracked,
        Inferred = 3,
    }

    private void IsInferred(object sender, AllFramesReadyEventArgs e)
    {
        handright = first1.Joints[JointType.HandRight];
        handleft = first1.Joints[JointType.HandLeft];
        footright = first1.Joints[JointType.FootRight];
        footleft = first1.Joints[JointType.FootLeft];
        ankleleft = first1.Joints[JointType.AnkleLeft];
        ankleright = first1.Joints[JointType.AnkleRight];
        elbowleft = first1.Joints[JointType.ElbowLeft];
        elbowright = first1.Joints[JointType.ElbowRight];
        head = first1.Joints[JointType.Head];
        hipcenter = first1.Joints[JointType.HipCenter];
        hipleft = first1.Joints[JointType.HipLeft];
        hipright = first1.Joints[JointType.HipRight];
        shouldercenter = first1.Joints[JointType.ShoulderCenter];
        shoulderleft = first1.Joints[JointType.ShoulderLeft];
        shoulderright = first1.Joints[JointType.ShoulderRight];
        kneeleft = first1.Joints[JointType.KneeLeft];
        kneeright = first1.Joints[JointType.KneeRight];
        spine = first1.Joints[JointType.Spine];
        wristleft = first1.Joints[JointType.WristLeft];
        wristright = first1.Joints[JointType.WristRight];

        if (handleft.TrackingState == JointTrackingState.Inferred &
            handright.TrackingState == JointTrackingState.Inferred &
            head.TrackingState == JointTrackingState.Inferred &
            footleft.TrackingState == JointTrackingState.Inferred &
            footright.TrackingState == JointTrackingState.Inferred &
            ankleleft.TrackingState == JointTrackingState.Inferred &
            ankleright.TrackingState == JointTrackingState.Inferred &
            elbowleft.TrackingState == JointTrackingState.Inferred &
            elbowright.TrackingState == JointTrackingState.Inferred &
            hipcenter.TrackingState == JointTrackingState.Inferred &
            hipleft.TrackingState == JointTrackingState.Inferred &
            hipright.TrackingState == JointTrackingState.Inferred &
            shouldercenter.TrackingState == JointTrackingState.Inferred &
            shoulderleft.TrackingState == JointTrackingState.Inferred &
            shoulderright.TrackingState == JointTrackingState.Inferred &
            kneeleft.TrackingState == JointTrackingState.Inferred &
            kneeright.TrackingState == JointTrackingState.Inferred &
            spine.TrackingState == JointTrackingState.Inferred &
            wristleft.TrackingState == JointTrackingState.Inferred &
            wristright.TrackingState == JointTrackingState.Inferred)
        {
            inferred = true;
        }
      }
    }
  }

The code in your project (compiler error)

    using IsInferred;

    static bool Inferred = false;
    SkeletonInferred inferred = new SkeletonInferred(Inferred);
    static Skeleton first1 = new Skeleton();

    Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton =>  skeleton.TrackingState == SkeletonTrackingState.Inferred);

Good luck, I hope this helps you get going in the right direction or helps you at all!

Here's my code that you asked for. Yes it is from the Skeletal Tracking Fundamentals, but this code was here and I didn't want to start a new project with most of the same stuff. Enjoy!

Code

     // (c) Copyright Microsoft Corporation.
     // This source is subject to the Microsoft Public License (Ms-PL).
    // Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details.
    // All other rights reserved.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Coding4Fun.Kinect.Wpf; 

namespace SkeletalTracking
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    bool closing = false;
    const int skeletonCount = 6; 
    Skeleton[] allSkeletons = new Skeleton[skeletonCount];

    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
        kinectSensorChooser1.KinectSensorChanged += new  DependencyPropertyChangedEventHandler(kinectSensorChooser1_KinectSensorChanged);

    }

    void kinectSensorChooser1_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        KinectSensor old = (KinectSensor)e.OldValue;

        StopKinect(old);

        KinectSensor sensor = (KinectSensor)e.NewValue;

        if (sensor == null)
        {
            return;
        }




        var parameters = new TransformSmoothParameters
        {
            Smoothing = 0.3f,
            Correction = 0.0f,
            Prediction = 0.0f,
            JitterRadius = 1.0f,
            MaxDeviationRadius = 0.5f
        };
        sensor.SkeletonStream.Enable(parameters);

        //sensor.SkeletonStream.Enable();

        sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady);
        sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); 
        sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

        try
        {
            sensor.Start();
        }
        catch (System.IO.IOException)
        {
            kinectSensorChooser1.AppConflictOccurred();
        }
    }

    void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {
        if (closing)
        {
            return;
        }

        //Get a skeleton
        Skeleton first =  GetFirstSkeleton(e);

        if (first == null)
        {
            return; 
        }



        //set scaled position
        //ScalePosition(headImage, first.Joints[JointType.Head]);
        ScalePosition(leftEllipse, first.Joints[JointType.HandLeft]);
        ScalePosition(rightEllipse, first.Joints[JointType.HandRight]);
        ScalePosition(leftknee, first.Joints[JointType.KneeLeft]);
        ScalePosition(rightknee, first.Joints[JointType.KneeRight]);

        GetCameraPoint(first, e); 

    }

    void GetCameraPoint(Skeleton first, AllFramesReadyEventArgs e)
    {

        using (DepthImageFrame depth = e.OpenDepthImageFrame())
        {
            if (depth == null ||
                kinectSensorChooser1.Kinect == null)
            {
                return;
            }


            //Map a joint location to a point on the depth map
            //head
            DepthImagePoint headDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.Head].Position);
            //left hand
            DepthImagePoint leftDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandLeft].Position);
            //right hand
            DepthImagePoint rightDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandRight].Position);

            DepthImagePoint rightKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeRight].Position);

            DepthImagePoint leftKnee =
                depth.MapFromSkeletonPoint(first.Joints[JointType.KneeLeft].Position);


            //Map a depth point to a point on the color image
            //head
            ColorImagePoint headColorPoint =
                depth.MapToColorImagePoint(headDepthPoint.X, headDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //left hand
            ColorImagePoint leftColorPoint =
                depth.MapToColorImagePoint(leftDepthPoint.X, leftDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //right hand
            ColorImagePoint rightColorPoint =
                depth.MapToColorImagePoint(rightDepthPoint.X, rightDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint leftKneeColorPoint =
                depth.MapToColorImagePoint(leftKnee.X, leftKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);

            ColorImagePoint rightKneeColorPoint =
                depth.MapToColorImagePoint(rightKnee.X, rightKnee.Y,
                ColorImageFormat.RgbResolution640x480Fps30);



            //Set location
            CameraPosition(headImage, headColorPoint);
            CameraPosition(leftEllipse, leftColorPoint);
            CameraPosition(rightEllipse, rightColorPoint);


            Joint LEFTKNEE = first.Joints[JointType.KneeLeft];
            Joint RIGHTKNEE = first.Joints[JointType.KneeRight];

            if ((LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
            LEFTKNEE.TrackingState == JointTrackingState.Tracked) &&
            (RIGHTKNEE.TrackingState == JointTrackingState.Tracked ||
            RIGHTKNEE.TrackingState == JointTrackingState.Inferred))
            {
                CameraPosition(rightknee, rightKneeColorPoint);
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (LEFTKNEE.TrackingState == JointTrackingState.Inferred ||
                    LEFTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(leftknee, leftKneeColorPoint);
            }

            else if (RIGHTKNEE.TrackingState == JointTrackingState.Inferred ||
                    RIGHTKNEE.TrackingState == JointTrackingState.Tracked)
            {
                CameraPosition(rightknee, rightKneeColorPoint);
            }
        }        
    }


    Skeleton GetFirstSkeleton(AllFramesReadyEventArgs e)
    {
        using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame())
        {
            if (skeletonFrameData == null)
            {
                return null; 
            }


            skeletonFrameData.CopySkeletonDataTo(allSkeletons);

            //get the first tracked skeleton
            Skeleton first = (from s in allSkeletons
                                     where s.TrackingState == SkeletonTrackingState.Tracked
                                     select s).FirstOrDefault();

            return first;

        }
    }

    private void StopKinect(KinectSensor sensor)
    {
        if (sensor != null)
        {
            if (sensor.IsRunning)
            {
                //stop sensor 
                sensor.Stop();

                //stop audio if not null
                if (sensor.AudioSource != null)
                {
                    sensor.AudioSource.Stop();
                }


            }
        }
    }

    private void CameraPosition(FrameworkElement element, ColorImagePoint point)
    {
        //Divide by 2 for width and height so point is right in the middle 
        // instead of in top/left corner
        Canvas.SetLeft(element, point.X - element.Width / 2);
        Canvas.SetTop(element, point.Y - element.Height / 2);

    }

    private void ScalePosition(FrameworkElement element, Joint joint)
    {
        //convert the value to X/Y
        //Joint scaledJoint = joint.ScaleTo(1280, 720); 

        //convert & scale (.3 = means 1/3 of joint distance)
        Joint scaledJoint = joint.ScaleTo(1280, 720, .3f, .3f);

        Canvas.SetLeft(element, scaledJoint.Position.X);
        Canvas.SetTop(element, scaledJoint.Position.Y); 

    }


    private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
    {
        closing = true; 
        StopKinect(kinectSensorChooser1.Kinect); 
    }

    private void kinectDepthViewer1_Loaded(object sender, RoutedEventArgs e)
    {

    }

   }
}

XAML

<Window x:Class="SkeletalTracking.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="600" Width="800" Loaded="Window_Loaded" 
    xmlns:my="clr-namespace:Microsoft.Samples.Kinect.WpfViewers;assembly=Microsoft.Samples.Kinect.WpfViewers" 
    Closing="Window_Closing" WindowState="Maximized">       
<Canvas Name="MainCanvas">
    <my:KinectColorViewer Canvas.Left="0" Canvas.Top="0" Width="640" Height="480" Name="kinectColorViewer1" 
                          Kinect="{Binding ElementName=kinectSensorChooser1, Path=Kinect}" />
    <Ellipse Canvas.Left="0" Canvas.Top="0" Height="50" Name="leftEllipse"  Width="50" Fill="#FF4D298D" Opacity="1" Stroke="White" />
    <Ellipse Canvas.Left="100" Canvas.Top="0" Fill="#FF2CACE3" Height="50" Name="rightEllipse" Width="50" Opacity="1" Stroke="White" />
    <my:KinectSensorChooser Canvas.Left="250" Canvas.Top="380" Name="kinectSensorChooser1" Width="328" />
    <Image Canvas.Left="66" Canvas.Top="90" Height="87" Name="headImage" Stretch="Fill" Width="84" Source="/SkeletalTracking;component/c4f-color.png" />
    <Ellipse Canvas.Left="283" Canvas.Top="233" Height="23" Name="leftknee" Stroke="Black" Width="29" />
    <Ellipse Canvas.Left="232" Canvas.Top="233" Height="23" Name="rightknee" Stroke="Black" Width="30" />
</Canvas>

Here's a picture just to show how off Kinect can be sometimes:

Hint: Notice how only my arm is detected and part of the background

Kinect sideways skeleton tracking

3 Answers

Edit 1

Code

XAML