Efficient Kinect Depth Image (or large Array) Filtering

Question

I am working with the Kinect SDK and I am trying to filter the depth image data in two ways:

Remove all depths that are not associated with a player
Remove all depths that are greater than a given depth (calculated form the position of a players wrists)

The result of which is essentially to show only the parts of a players body which are less than a certain depth from the sensor.

Whilst the code below does what i want it to do, its not proving very good when i run performance analysis, so I am looking for ways in which it could be improved.

The basic issue is that the array contains 307200 values (as the depth image size is 640x480), and I am trying to get this method called around 30 times a second.

Does anyone have any pointers as to how this could be done more efficiently? The code also uses EmguCV libraries in other parts and I have tinkered with using the cvInvokeThreshold method but it didn't seeem to work as well as this code...

If you need any more information, let me know.

Many Thanks,

Dave McB

public static byte[] GetDepths(byte[] depths, DepthImagePixel[] depthPixels, int width, int height, int threshold)
    {

        Parallel.For(0, width, i =>
        {
            for (int j = 0; j < height; j++)
            {
                //Have to calculate the index we are working on using i and j
                int rawDepthDataIndex = i * height + j;

                //gets the depth and player values
                short depth = depthPixels[rawDepthDataIndex].Depth;
                short player = depthPixels[rawDepthDataIndex].PlayerIndex;

                if (player > 0 && depth < threshold)
                    depths[rawDepthDataIndex] = (byte)depth;
                else
                    depths[rawDepthDataIndex] = 0;
            }


        });
        return depths;
    }

I can't think of any optimizations other than Parallel.For which you already have, but I do have a small comment in the code. Shouldn't you have: Parallel.For(0, height, i => { for (int j = 0; j < width; j++) { int rawDepthDataIndex = i * height + j; ? — Julia Schwarz
My reason for that way round was that the width is greater (680) so having the larger index in the parallel for would lead to greater performance than a smaller index, is that the wrong way round? — DMcB1888
Okay this makes sense. I usually process images row-wise because the images are stored row-wise. Have you tried the other way around? One suggestion to make the code faster is to downsample the image. Also, try profiling the code and see which instruction is slowest. — Julia Schwarz
So far i've redcued the image resolution and have now sucessfully shifted the threshold into the emgucv threshold function, go to run the profiling again tomorrow as i can't run it from home. When I last profiled the code the if (player > 0 && depth < threshold) was the line the caught over 60% over the profiles all the time. With the threshold now done in emgucv that line now reads if (player > 0) so profiling tomorrow should be good! — DMcB1888

user2373777 user2373777 · Accepted Answer · 2013-05-11T21:34:35

Hi i have a problema to depth image and cut .

I use this code

Using depthFrame As DepthImageFrame = e.OpenDepthImageFrame()
            If depthFrame IsNot Nothing Then
                ' Copy the pixel data from the image to a temporary array
                Me.depthPixels = Me.smoothingFilter.CreateFilteredDepthArray(Me.depthPixels, depthFrame.Width, depthFrame.Height)

                depthFrame.CopyDepthImagePixelDataTo(Me.depthPixels)

                depthReceived = True
            End If
End Using

Efficient Kinect Depth Image (or large Array) Filtering

1 Answers