Kinect SDK 1.7: Mapping Joint/Cursor Coordinates to screen Resolution

Question

Unfortunately I'm still struggling a little with the new Kinect SDK 1.7. This problem is actually in the same context as "finding events via reflection c#": the Click event (however, it is not necessary for understanding this problem).

My Problem is simple: if I have my right hand controlling the cursor (the "new" Kinect HandPointer) and it's in the upper left corner of the screen I want it to return the Coordinates (0,0). If the cursor is in the lower right corner the coordinates should be (1920,1080) respectively the current screen resolution.

The new SDK has so-called PhysicalInteractionZones (PIZ) for each HandPointer (up to 4) which move with the HandPointers and have values from (for upper-left) 0.0 to (for lower-right) 1.0. Which basically means, I can't use them for mapping to the screen since they are changing dynamically according to the users movement in front of the Kinect. At least, I was unable to find a way to make that work.

I then tried it via SkeletonStream: the coordinates for the right hand are tracked and as soon as a click gesture is registered, the Click-Event triggers at this specific point. I tried it with the following code:

private void ksensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    {
        using (SkeletonFrame frame = e.OpenSkeletonFrame())
        {
            if (frame != null)
            {
                frame.CopySkeletonDataTo(this._FrameSkeletons);
                var accelerometerReading = 
                    Settings.Instance.ksensor.AccelerometerGetCurrentReading();
                ProcessFrame(frame);
                _InteractionStream.ProcessSkeleton(_FrameSkeletons,
                    accelerometerReading, frame.Timestamp);
            }
        }
    }

private void ProcessFrame(ReplaySkeletonFrame frame)
    {
        foreach (var skeleton in frame.Skeletons)
        {
            if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
                continue;
            foreach (Joint joint in skeleton.Joints)
            {
                if (joint.TrackingState != JointTrackingState.Tracked)
                    continue;
                if (joint.JointType == JointType.HandRight)
                {
                    _SwipeGestureDetectorRight.Add(joint.Position,
                        Settings.Instance.ksensor);
                    _RightHand = GetPosition(joint);
                    myTextBox.Text = _RightHand.ToString();
                }
                if (joint.JointType == JointType.HandLeft)
                {
                    _SwipeGestureDetectorLeft.Add(joint.Position, 
                        Settings.Instance.ksensor);
                    _LeftHand = GetPosition(joint);
                }
            }
        }
    }

The auxialiary GetPosition method is defined as the following:

private Point GetPosition(Joint joint)
    {
        DepthImagePoint point = 
            Settings.Instance.ksensor.CoordinateMapper.MapSkeletonPointToDepthPoint(joint.Position, Settings.Instance.ksensor.DepthStream.Format);
        point.X *= 
            (int)Settings.Instance.mainWindow.ActualWidth / Settings.Instance.ksensor.DepthStream.FrameWidth;
        point.Y *= 
            (int)Settings.Instance.mainWindow.ActualHeight / Settings.Instance.ksensor.DepthStream.FrameHeight;

        return new Point(point.X, point.Y);
    }

As soon as the click gesture is detected a simple invokeClick(_RightHand) is called and performs the click. The click itself is working perfectly fine (thanks again to the people who answered on that issue). What is not working so far is the mapping of the coordinates since I only get coordinates from

x-Axis: 900 - 1500 (from left to right) y-Axis: 300 - 740 (from top to bottom)

And these coordinates even vary with each time I try to reach one specific point on the screen by 100 or 200 pixels, e.g. the left-hand side of the screen is 900 at first but when I move my hand out of the range of the Kinect (behind my back or under the table) and repeat the movement towards the left-hand side I suddenly get coordinates of 700 or something around that. I even tried the ScaleTo methods from Coding4Fun.Kinect.Wpf (ScaleTo(1920,1080) respectively ScaleTo(SystemParameters.PrimaryScreenWidth,SystemParameters.PrimaryScreenHight)) but that just gave me crazy coordinates in the x:300000, y:-100000 or 240000...and I'm running out of ideas so I hope someone out there has one for me or even a solution for this.

Sorry for the long text but I've tried to be as specific as I could be. Thanks in advance for any help!

Answer 1

The InteractionHandPointer class contains the screen coordinates for the hand. I adapted this code from the SDK demos; there are many hoops to jump through!:

this.sensor.DepthFrameReady += this.Sensor_DepthFrameReady;
this.interaction = new InteractionStream(sensor, new InteractionClient());
this.interaction.InteractionFrameReady += interaction_InteractionFrameReady;

...

private void Sensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
    using (var frame = e.OpenDepthImageFrame())
    {
        if (frame != null)
        {
            try
            {
                interaction.ProcessDepth(frame.GetRawPixelData(), frame.Timestamp);
            }
            catch (InvalidOperationException) { }
        }
    }
}

private void interaction_InteractionFrameReady(object sender, InteractionFrameReadyEventArgs e)
{
    using (var frame = e.OpenInteractionFrame())
    {
        if (frame != null)
        {
            if ((interactionData == null) || 
                (interactionData.Length !== InteractionStream.FrameUserInfoArrayLength))
            {
                interactionData = new UserInfo[InteractionStream.FrameUserInfoArrayLength];
            }
            frame.CopyInteractionDataTo(interactionData);

            foreach (var ui in interactionData)
            {
                foreach (var hp in ui.HandPointers)
                {
                    // Get screen coordinates
                    var screenX = hp.X * DisplayWidth;
                    var screenY = hp.Y * DisplayHeight;

                    // You can also access IsGripped, IsPressed etc.
                }
            }
        }
    }
}

public class InteractionClient: IInteractionClient
{
    public InteractionInfo GetInteractionInfoAtLocation(
        int skeletonTrackingId,
        InteractionHandType handType,
        double x, double y)
    {
        return new InteractionInfo();
    }
}

Answer 2

A colleague of mine was trying to map hand coords to screen coords. He did it successfully by some simple math; but the result is not favorable since a small number of pixels (hand area) would be mapped to a much larger number of pixels, this would cause the cursor to be very inaccurate and jump around with the natural small shakes of your hand when you are holding it up in the air, or the joint recognition inaccuracies.

Instead of this method, I would recommend what has been done in laptop trackpads: if you move your hand very slowly on the trackpad, the cursor would only move a very small range. But if you move your finger very fast, the cursor travels a bigger distance. So, if for example the hand coords move 100px slowly, you can move the cursor 100px on the screen, but if the same 100px movement of the hand has happened fast, you can move the cursor say 300px. This is independent of any mapping between hand coord space and screen coord space.

Answer 3

Another way to solve this problem is by using the code provided in an example in kinect sdk toolkit v1.8 named "Skeleton Basics - WPF" then you have a function already set to receive a skeleton point and return a point like so:

    private Point SkeletonPointToScreen(SkeletonPoint skelpoint)
    {
        // Convert point to depth space.  
        // We are not using depth directly, but we do want the points in our 640x480 output resolution.
        DepthImagePoint depthPoint = this.sensor.CoordinateMapper.MapSkeletonPointToDepthPoint(skelpoint, DepthImageFormat.Resolution640x480Fps30);
        return new Point(depthPoint.X, depthPoint.Y);

    }