Google Speech Recognition timeout

Question

I am developing an Android Application that is based around Speech Recognition.

Until today everything has been working fine and in a timely manner, e.g. I would start my speech recogniser, speak, and within 1 or 2 seconds max the application received the results.

It was a VERY acceptable user experience.

Then today I now have to wait for ten or more seconds before the recognition results are available.

I have tried setting the following EXTRAS, none of which make any discernible difference

RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS
RecognizerIntent.EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS
RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS

I have been continually changing my application, however none of these changes were related to the speech recogniser.

Is there any method I can employ to reduce the time between the speech recogniser switching from onBeginningOfSpeech() to onResults()?

Heres an example of how long it takes

07-01 17:50:20.839 24877-24877/com.voice I/Voice: onReadyForSpeech()
07-01 17:50:21.614 24877-24877/com.voice I/Voice: onBeginningOfSpeech()
07-01 17:50:38.163 24877-24877/com.voice I/Voice: onEndOfSpeech()

Answer 1

EDIT - Has apparently been fixed in the August 2016 coming release You can test the beta to confirm.

This is a bug with the release of Google 'Now' V6.0.23.* and persists in the latest V6.1.28.*

Since the release of V5.11.34.* Google's implementation of the SpeechRecognizer has been plagued with bugs.

You can use this gist to replicate many of them.

You can use this BugRecognitionListener to work around some of them.

I have reported these directly to the Now team, so they are aware, but as yet, nothing has been fixed. There is no external bug tracker for Google Now, as it's not part of AOSP, so nothing you can star I'm afraid.

The most recent bug you detail pretty much makes their implementation unusable, as you correctly point out, the parameters to control the speech input timings are ignored. Which according to the documentation:

Additionally, depending on the recognizer implementation, these values may have no effect.

is something we should expect......

The recognition will continue indefinitely if you don't speak or make any detectable sound.

I'm currently creating a project to replicate this new bug and all of the others, which I'll forward on and link here shortly.

EDIT - I was hoping I could create a workaround that used the detection of partial or unstable results as the trigger to know that the user was still speaking. Once they stopped, I could manually call recognizer.stopListening() after a set period of time.

Unfortunately, stopListening() is broken too and doesn't actually stop the recognition, therefore there is no workaround to this.

Attempts around the above, of destroying the recognizer and relying only on the partial results up until that point (when destroying the recognizer onResults() is not called) failed to produce a reliable implementation, unless you're simply keyword spotting.

There is nothing we can do until Google fix this. Your only outlet is to email [email protected] reporting the problem and hope that the volume they receive gives them a nudge.....

Answer 2

NOTE! this works only in online mode. Enable dictation mode and disable partial results:

intent.putExtra("android.speech.extra.DICTATION_MODE", true);
intent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, false);

In dictation mode speechRecognizer would still call onPartialResults() however you should treat the partials as final results.

Answer 3

UPDATE:

Just in case if anyone is having trouble in setting up the speech recognition, you can use Droid Speech library which I built to overcome the speech time out issue in android.

---

My app was entirely dependent upon the voice recognition feature and Google has dropped a bomb. Going by the look of things, I believe this wouldn't be fixed at least in the near future.

For the time being, I did find a solution to have the google voice recognition deliver the speech results as intended.

Note: This approach slightly varies from the above mentioned solutions.

The main purpose of this method is to make sure the entire words uttered by the user is caught at onPartialResults().

In normal cases if a user speaks more than a single word at a given instance the response time is too quick and partial results will more often than not get only the first word and not the complete result.

So to make sure every single word is caught at onPartialResults() a handler is introduced to check the user pause delay and then filter the results. Also note the result array from onPartialResults() will more often than not have only a single item.

SpeechRecognizer userSpeech = SpeechRecognizer.createSpeechRecognizer(this);

Intent speechIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
speechIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
speechIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, this.getPackageName());
speechIntent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true);
speechIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, ModelData.MAX_VOICE_RESULTS);

Handler checkForUserPauseAndSpeak = new Handler(); 
Boolean speechResultsFound = false;

userSpeech.setRecognitionListener(new RecognitionListener(){

    @Override
    public void onRmsChanged(float rmsdB)
    {
        // NA
    }

    @Override
    public void onResults(Bundle results)
    {
        if(speechResultsFound) return;

        speechResultsFound = true;

        // Speech engine full results (Do whatever you would want with the full results)
    }

    @Override
    public void onReadyForSpeech(Bundle params)
    {
        // NA
    }

    @Override
    public void onPartialResults(Bundle partialResults)
    {
        if(partialResults.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).size() > 0 &&
                partialResults.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).get(0) != null &&
                !partialResults.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).get(0).trim().isEmpty())
        {
            checkForUserPauseAndSpeak.removeCallbacksAndMessages(null);
            checkForUserPauseAndSpeak.postDelayed(new Runnable()
            {
                @Override
                public void run()
                {
                    if(speechResultsFound) return;

                    speechResultsFound = true;

                    // Stop the speech operations
                    userSpeech.destroy();

                    // Speech engine partial results (Do whatever you would want with the partial results)

                }

            }, 1000);
        }
    }

    @Override
    public void onEvent(int eventType, Bundle params)
    {
        // NA
    }

    @Override
    public void onError(int error)
    {
        // Error related code
    }

    @Override
    public void onEndOfSpeech()
    {
        // NA
    }

    @Override
    public void onBufferReceived(byte[] buffer)
    {
        // NA
    }

    @Override
    public void onBeginningOfSpeech()
    {
        // NA
    }
});

userSpeech.startListening(speechIntent);

Answer 4

The best work around solution I found (until google fixes the bug) was to go in to the Google App App info and then click on "Uninstall Updates" button. This will remove all updates done to this app which has direct affect on the speech recognizer, basically returning it to factory.

**Probably a good idea to stop automatic updates until we know it's fixes. ***Note: this is a solution only for developers, obviously if you have an app in the store this will not help you. Sorry...

Answer 5

UPDATE: As of my testing today, this bug seems to have been resolved finally and this is no longer necessary. Leaving it in case it gets broke again in the future. From my tests, the speech timeout is working normally.

Ok, I know this is VERY UGLY, but it seems to work using onPartialResults (I understand the gotchas with onPartialResults but I've tried this a few times and it's something until Google fixes this ridiculous bug!) I haven't exhaustively tested it yet (I will and post back results as I will be using this in an app) but I was desperate for a solution. Basically, I'm using onRmsChanged to trigger that user is done speaking, assuming that when the RmsDb falls below peak and no onPartialResults for 2 seconds, we're done.

The one thing I don't like about this is destroying SR makes a double uh-oh beep. FWIW and YMMV. Please post any improvements!

NOTE: If you are going to use this repeatedly, don't forget to reset bBegin and fPeak! Also you will need to recreate SR (either onStartCommand or stop and start service.)

import android.app.Service;
import android.content.Intent;
import android.os.Bundle;
import android.os.IBinder;
import android.speech.RecognitionListener;
import android.speech.RecognizerIntent;
import android.speech.SpeechRecognizer;
import android.support.annotation.Nullable;
import android.util.Log;

import java.util.ArrayList;

public class SpeechToTextService extends Service {

    private String TAG = "STT";

    float fPeak;
    boolean bBegin;
    long lCheckTime;
    long lTimeout = 2000;

    @Override
    public void onCreate() {
        super.onCreate();

        bBegin = false;
        fPeak = -999; //Only to be sure it's under ambient RmsDb.

        final SpeechRecognizer sr = SpeechRecognizer.createSpeechRecognizer(getApplicationContext());
        sr.setRecognitionListener(new RecognitionListener() {

            @Override
            public void onReadyForSpeech(Bundle bundle) {
                Log.i(TAG, "onReadyForSpeech");
            }

            @Override
            public void onBeginningOfSpeech() {
                bBegin = true;
                Log.i(TAG, "onBeginningOfSpeech");
            }

            @Override
            public void onRmsChanged(float rmsDb) {
                if(bBegin) {
                    if (rmsDb > fPeak) {
                        fPeak = rmsDb;
                        lCheckTime = System.currentTimeMillis();
                    }
                    if (System.currentTimeMillis() > lCheckTime + lTimeout) {
                        Log.i(TAG, "DONE");
                        sr.destroy();
                    }
                }
                //Log.i(TAG, "rmsDB:"+rmsDb);
            }

            @Override
            public void onBufferReceived(byte[] buffer) {
                Log.i(TAG, "onBufferReceived");
            }

            @Override
            public void onEndOfSpeech() {
                Log.i(TAG, "onEndOfSpeech");
            }

            @Override
            public void onError(int error) {
                Log.i(TAG, "onError:" + error);
            }

            @Override
            public void onResults(Bundle results) {

                ArrayList data = results.getStringArrayList(
                        SpeechRecognizer.RESULTS_RECOGNITION);

                String sTextFromSpeech;
                if (data != null) {
                    sTextFromSpeech = data.get(0).toString();
                } else {
                    sTextFromSpeech = "";
                }
                Log.i(TAG, "onResults:" + sTextFromSpeech);
            }

            @Override
            public void onPartialResults(Bundle bundle) {

                lCheckTime = System.currentTimeMillis();
                ArrayList data = bundle.getStringArrayList(
                        SpeechRecognizer.RESULTS_RECOGNITION);

                String sTextFromSpeech;
                if (data != null) {
                    sTextFromSpeech = data.get(0).toString();
                } else {
                    sTextFromSpeech = "";
                }
                Log.i(TAG, "onPartialResults:" + sTextFromSpeech);
            }

            @Override
            public void onEvent(int eventType, Bundle params) {

                Log.i(TAG, "onEvent:" + eventType);
            }
        });

        Intent iSRIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        iSRIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        iSRIntent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true);
        iSRIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, getPackageName());
        iSRIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, "en-US");
        iSRIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, "en-US");
        sr.startListening(iSRIntent);
    }

    @Nullable
    @Override
    public IBinder onBind(Intent intent) {
        return null;
    }
}

Answer 6

Solution for offline only:

I have encountered the same issue (Android system took 25 seconds to yield the transcript of the speech via onPartialResults() after onEndOfSpeech() is triggered.

I have tried the following code and it worked:

Intent.putExtra
(
    RecognizerIntent.EXTRA_PREFER_OFFLINE,
    true
);

This solution works for my app and it may work for you if you do not use the online mode (I downloaded the language package via the phone setting).