Using Kinect Skeleton ID in Speech Recognition

1
votes

I am doing a project around the Kinect. Basically, I have a database of audio files for a number of users.

Out of those users, only one will be standing in front of the Kinect (in its range). The Kinect will be tracking the skeleton, giving him an ID, and also I have managed to display only the tracked skeleton's head. At the same time, I have applied a speech recognition code, so the user will say key words and actions will happen based on those key words. PS: The actions will happen only if the user's voice is found "matched" with one of the audio files in the data base.

Okay, so far so good. All of the above is working relatively okay. Here's the problem: Let's imagine a user standing in front of the Kinect, a "Verified user", at the same time another user walks in, also verified, and says one of the key words. Now when that second user walks in, he will be given a skeleton track ID.

Is there a way that I can pass the track ID of the first skeleton to the speech recognition code so that the actions will happen when the key words are only said by the first user - based on his track ID -?

Thanks.

1
kinecttrackingspeech

1 Answers

0
votes

You can use the audio source angle to determine where the sound is coming from. If you have that information and the user's skeleton position, you can tell which user is speaking. Note that this will give you the direction of the loudest audio... wich will be fine assuming your users are not speaking at the same time.

https://msdn.microsoft.com/en-us/library/microsoft.kinect.kinectaudiosource.soundsourceangle.aspx

That's of course assuming you're using Kinect 1 (SDK 1.8)... if you're using Kinect 2, you can directly ask the body traking ID of the user who is speaking. Just ask the AudioBeam for a frame, ask the frame for a subframe and ask the subframe for the correlations, then you have this info:

https://msdn.microsoft.com/en-us/library/microsoft.kinect.kinect.iaudiobodycorrelation.get_bodytrackingid.aspx https://msdn.microsoft.com/en-us/library/windowspreview.kinect.audiobodycorrelation.bodytrackingid.aspx