Stiching multiple textures/frames together in OpenGL using a Kinect

Question

I came across the following situation:

I have a Kinect camera and I keep taking frames (but they are stored only when the user presses a key).
I am using the freenect library in order to retrieve the depth and the color of the frame (I am no interested in skeleton tracking or something like that).
For a single frame I am using the glpclview example that comes with the freenect library

After retrieving the space data from the Kinect sensor, in the glpclview example, the current frame it is drawn like this:

glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(3, GL_SHORT, 0, xyz);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glTexCoordPointer(3, GL_SHORT, 0, xyz);

glEnable(GL_TEXTURE_2D);
glBindTexture(GL_TEXTURE_2D, gl_rgb_tex);
glTexImage2D(GL_TEXTURE_2D, 0, 3, 640, 480, 0, GL_RGB, GL_UNSIGNED_BYTE, rgb);

glPointSize(2.0f);
glDrawElements(GL_POINTS, 640*480, GL_UNSIGNED_INT, indices);

where

static unsigned int indices[480][640];
static short xyz[480][640][3];
char *rgb = 0;
short *depth = 0;

where:

rgb is the color information for the current frame
depth is the depth information for the current frame
xyz is constructed as :

xyz[i][j][0] = j xyz[i][j]3 = i xyz[i][j]4 = depth[i*640+j]
indices is (I guess only) array that keeps track of the rgb/depth data and is constructed as:

indices[i][j] = i*640+j

So far, so good, but now I need to render more that just one frame (some of them rotated and translated with a certain angle/offsets). How can I do this? I'ved tried to increase the size of the arrays and keep reallocationg memory for each new frame, but how can I render them? Should I change this current line to something else?

glTexImage2D(GL_TEXTURE_2D, 0, 3, 640, 480, 0, GL_RGB, GL_UNSIGNED_BYTE, rgb)

If so, to what values should I change 640 and 480 since now xyz and rgb is a contiguos pointer of 640x480x(number of frames)?

To get a better ideea, I am trying to get something similar to this in the end (except the robot :D ).

If somewone has a better ideea, hint anything on how I should approach this problem, please let me know.

mankoff mankoff · Accepted Answer · 2012-02-13T02:59:50

It isn't as simple as allocating a bigger array.

If you want to stitch together multiple point-clouds to make a bigger map, you should look into the SLAM algorithms (that is what they are running in the video your link to). You can find many implementations at http://openslam.org. You might also look into an ICP algorithm (Iterative Closest Point) and the KinectFusion from Microsoft (and the open source KinFu implementation from PCL).

Stiching multiple textures/frames together in OpenGL using a Kinect

1 Answers