I can't get kinect sdk to do speech recognition and track skeletal data at the sime time

Question

I' ve a program in witch I enabled speech recognition with..

        RecognizerInfo ri = GetKinectRecognizer();

        speechRecognitionEngine = new SpeechRecognitionEngine(ri.Id);

        // Create a grammar from grammar definition XML file.
        using (var memoryStream = new MemoryStream(Encoding.ASCII.GetBytes(fileContent)))
        {
            var g = new Grammar(memoryStream);
            speechRecognitionEngine.LoadGrammar(g);
        }

        speechRecognitionEngine.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(speechEngine_SpeechRecognized);
        speechRecognitionEngine.SpeechRecognitionRejected += new EventHandler<SpeechRecognitionRejectedEventArgs>(speechEngine_SpeechRecognitionRejected);

speechRecognitionEngine.SetInputToAudioStream( sensor.AudioSource.Start(), new SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));

        speechRecognitionEngine.RecognizeAsync(RecognizeMode.Multiple);

..

all'is working fine and SpeechRecognized event get fired correctly..

The problem is, when I anable skeletal data tracking,

          sensor.SkeletonStream.Enable();
          sensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
          sensor.SkeletonFrameReady += sensor_SkeletonFrameReady;

speech recognition stops working ...

can I get your help?

Thank you so much!

What are the system specs of your computer? Both the speech recognized and the skeleton tracker take a good chunk of processing power. You may be taxing your CPU too much. — Nicholas Pappas
Are you just enabling tracking or are you doing something else with the Skeleton frame which could be hanging your app? — Robben_Ford_Fan_boy

Atul Verma Atul Verma · Accepted Answer · 2012-12-29T12:30:42

Audio is not processed if skeleton stream is enabled after starting audio capture Due to a bug, enabling or disabling the SkeletonStream will stop the AudioSource stream returned by the Kinect sensor. The following sequence of instructions will stop the audio stream: kinectSensor.Start(); kinectSensor.AudioSource.Start(); // --> this will create an audio stream kinectSensor.SkeletonStream.Enable(); // --> this will stop the audio stream as an undesired side effect

The workaround is to invert the order of the calls or to restart the AudioSource after changing SkeletonStream status.

        Workaround #1 (start audio after skeleton):
        kinectSensor.Start();
        kinectSensor.SkeletonStream.Enable();
        kinectSensor.AudioSource.Start();

        Workaround #2 (restart audio after skeleton):
        kinectSensor.Start();
        kinectSensor.AudioSource.Start(); // --> this will create an audio stream
        kinectSensor.SkeletonStream.Enable(); // --> this will stop the audio stream as an undesired side effect
        kinectSensor.AudioSource.Start(); // --> this will create another audio stream

Source - http://msdn.microsoft.com/en-us/library/jj663798.aspx

I can't get kinect sdk to do speech recognition and track skeletal data at the sime time

1 Answers