DialogFlow StreamingDetectIntentResponse doesn't return ResponseId, QueryText, Transcript or anything except LanguageCode

Question

I have successfully used Grpc in Unity and sent request to Dialog flow and received response. You can check the details here

However the whole returned result is the following only

{ "queryResult": { "languageCode": "ja" } }

The expected response id, query text, etc are not returned. When testing in console.dialogflow.com I get the following result

{ "responseId": "cdf8003e-6599-4a28-9314-f4462c36e21b", "queryResult": { "queryText": "おはようございます", "speechRecognitionConfidence": 0.92638445, "languageCode": "ja" } }

However when I tried in console.dialogflow.com and didn't say anything I got

{ "queryResult": { "languageCode": "ja" } }

So perhaps the InputAudio encoding is wrong somehow.

Here's how I do it

var serializedByteArray = convertToBytes(samples);
request.InputAudio = Google.Protobuf.ByteString.CopyFrom(serializedByteArray);

And convert to bytes is like the following

public static byte[] convertToBytes(float[] audio)
{
    List<byte> bytes = new List<byte>();

    foreach (float audioI in audio) {
        bytes.AddRange(BitConverter.GetBytes(audioI));
    }

    return bytes.ToArray();
}

The audio source is define as follows where sampleRate is 16000

audioSource.clip = Microphone.Start(null, true, 30, sampleRate);

I made sure to set sample rate hz properly.

queryInput.AudioConfig.SampleRateHertz = sampleRate;

Edit:

I have logged the recorded bytes from unity to a file (have all the bytes streamed appended together) and have written a console application to test the binary generated but using DetectIntent rather than streaming detect intent.

GoogleCredential credential = GoogleCredential.FromJson(privateKey);

var url = "dialogflow.googleapis.com";

Grpc.Core.Channel channel = new Grpc.Core.Channel(url, credential.ToChannelCredentials());


var client = SessionsClient.Create(channel);


CallOptions options = new CallOptions();

DetectIntentRequest detectIntentRequest = new DetectIntentRequest();
detectIntentRequest.Session = "projects/projectid/agent/sessions/" + "detectIntent";
QueryInput queryInput = new QueryInput();
queryInput.AudioConfig = new InputAudioConfig();
queryInput.AudioConfig.LanguageCode = "ja";
queryInput.AudioConfig.SampleRateHertz = sampleRate;//must be between 8khz and 48khz
queryInput.AudioConfig.AudioEncoding = AudioEncoding.Linear16;

detectIntentRequest.QueryInput = queryInput;

detectIntentRequest.InputAudio = Google.Protobuf.ByteString.CopyFrom(File.ReadAllBytes("D:\\temp\\audio.bytes"));
 var response = client.DetectIntent(detectIntentRequest);
        Console.WriteLine(response.ToString());
        Console.WriteLine(response.ResponseId);
Console.Read();

I still get this (and empty response.ResponseId)

{ "queryResult": { "languageCode": "ja" } }

Thanks for advance.

Have you reproduced this in a console app as I've suggested? If so, the Unity aspect is irrelevant. Please show the code you're using - ideally as a complete console app. — Jon Skeet
@JonSkeet Thank you for your comment. Since audio format and encoding is a little bit unity specific it's hard to make a console app to test that but you gave me an idea. console.dialogflow.com has a handy test tool and I can even copy the curl command. So I'll do that but pass the audio encoded from unity instead of whatever in the console. I'm worried about the huge length of the curl command but I'll try to make it work for testing. — RamenTokyo
The audio format really shouldn't be Unity-specific - you'll need to use an audio format/encoding that Dialogflow supports, after all. Just record a file using Audacity or something similar, and you can try that from a console app. Again, this is all part of the diagnostic process - if you can get it working in a simple environment, that suggests the problem is what you're doing in Unity, not the Dialogflow API. You should be aiming to isolate the problem as far as you can. — Jon Skeet
Or if you can get the data from Unity, you can use that from a console app - that would be clearer than using curl, IMO. Fundamentally the approach is the same though: as Unity is an unsupported platform, take it out of the picture to start with. — Jon Skeet
@JonSkeet Just added an edit with the example console app. Thank you for advance. — RamenTokyo

RamenTokyo RamenTokyo · Accepted Answer · 2018-12-18T02:04:08

Finally found the answer. The way I converted the datasource float to linear16 byte array was obviously wrong. Here's the code that worked Credits to that post on unity forum.

https://forum.unity.com/threads/writing-audiolistener-getoutputdata-to-wav-problem.119295/#post-899142

public static byte[] convertToBytes(float[] dataSource)
{
    var intData = new Int16[dataSource.Length];
    //converting in 2 steps : float[] to Int16[], //then Int16[] to Byte[]

    var bytesData = new Byte[dataSource.Length * 2];
    //bytesData array is twice the size of
    //dataSource array because a float converted in Int16 is 2 bytes.

    var rescaleFactor = 32767; //to convert float to Int16

    for (var i = 0; i < dataSource.Length; i++)
    {
        intData[i] = (short)(dataSource[i] * rescaleFactor);
        var byteArr = new byte[2];
        byteArr = BitConverter.GetBytes(intData[i]);
        byteArr.CopyTo(bytesData, i * 2);
    }

    return bytesData;
}

DialogFlow StreamingDetectIntentResponse doesn't return ResponseId, QueryText, Transcript or anything except LanguageCode

Edit:

1 Answers