Apply Projector-Kinect calibration to Unity3d Camera

Question

I am developing an interactive art installation with multiple Kinect V2, a Projector and Unity3D.

Therefore, I used the technique described here: https://github.com/genekogan/KinectProjectorToolkit and implemented my own Calibration in Unity3D in C#.

The result of the Calibration is a DotNetMatrix with 11 entries.

With that Matrix, I can get the correctly mapped 2D Screen position of a 3D Point with the following method.

public static Vector2 convertKinectToProjector(Vector3 ksp)
{
    Vector3 kp = new Vector3(ksp.x * 1000, ksp.y * 1000, ksp.z * 1000);
    Vector2 mappedScreenPoint = new Vector2();
    float denom = (float)x.GetElement(8, 0) * kp.x + (float)x.GetElement(9, 0) * kp.y + (float)x.GetElement(10, 0) * kp.z + 1;
    mappedScreenPoint.x = Screen.width * ((float)x.GetElement(0, 0) * kp.x + (float)x.GetElement(1, 0) * kp.y + (float)x.GetElement(2, 0) * kp.z + (float)x.GetElement(3, 0)) / denom;
    mappedScreenPoint.y = Screen.height * ((float)x.GetElement(4, 0) * kp.x + (float)x.GetElement(5, 0) * kp.y + (float)x.GetElement(6, 0) * kp.z + (float)x.GetElement(7, 0)) / denom;

    return mappedScreenPoint;
}

With that solution, I can display Textures and other things correctly mapped on the GUI, but I also want to use the advantages of the 3D space.

So I'm looking for a way to get this calibration into the Unity3D camera. My aim is, that everything that the camera sees is correctly mapped from the 3D space to Screen space.

I've already setup the Unity3D camera. It has the correct Field of View(60) and position (0,0,0) like the Kinect camera. So if I generate a skeleton in Unity3D from the Kinect Body info everything matches like the real world sizes etc.

The missing part is the calibration itself.

I thought of two solutions:

a Camera Shader

or

Alter the Projection Matrix

Is there a way to implement that method for converting a 3D point to 2D into a shader, so that it will translate every Pixel to the right position.

Or can I simply change the Projection Matrix of the Unity Camera? The Projection Matrix in Unity3d is a 4*4 Matrix in a column-major. My Calibration Matrix is an 11*1 Matrix, is there a way to generate a 4*4 Matrix from that source?

Marush Kräftig Marush Kräftig · Accepted Answer · 2015-05-30T15:35:58

I used RoomAliveToolkit to get a projection matrix. My Unity project also interpretes the data and changes te projector so, thatthe camera is projecting from it´s perspective. But the skelet didn't fit the projection. I asked the microft guys and they gave me a hint. Be aware that kinect is using right handed coordinate system and unity left handed.

I don't know why your matrix is a 11*1. But if you like I can support you. Do you share your source on github or bitbucket?

Apply Projector-Kinect calibration to Unity3d Camera

2 Answers