Asking while using SSML in Dialogflow webhook

0
votes

I am trying to build an Actions on Google Agent via DialogFlow and keep getting errors when trying to ask the user a question while including ssml.

I have built the agent on DialogFlow, have logic implemented using the fulfillment webhook (implemented via the node module dialogflow-fulfillment) and have been able to test on DialogFlow successfully using the test console on the right side DialogFlow.

I therefore hooked up the DialogFlow Integrations to Google Assistant.

I first tried unsuccessfully:

const client = new WebhookClient({ req, res });
let qToSnd = 'Hi <break time=\"500ms\"/> Can I help you?';
let conv = client.conv();
conv.ask(qToSnd);
client.add(conv);

The above would work (not give errors) but would result in the question being asked while speaking out the <break> tag.

I have also tried:

conv.ask(
  new Text({
    text: _stripTags(qToSnd),
    ssml: qToSnd
}));

However, when I test this using the Actions on Google simulator I get the error message:

[Agent] isn't responding right now. Try again soon.

Digging into the logs viewer shows the following error message:

MalformedResponse: ErrorId: ... Failed to parse Dialogflow response into AppResponse because of invalid platform response. : Could not find a RichResponse or SystemIntent in the platform response for agentId: ... and intentId: ...

My fulfillment API is returning:

{
  "payload": {
    "google": {
      "expectUserResponse": true,
      "richResponse": {
        "items": [
          {
            "text": "Hi - Can I help you?",
            "ssml": "Hi <break time=\"500ms\"/> Can I help you?"
          }
        ]
      }
    }
  }
}

I will appreciate any pointers in the right direction.

2
actions-on-google
It may be helpful to post the actual code you used to generate the response - Nick Felker
@NickFelker - posting the actual code is a little hard. I will try to make a simple app to do that. Are you saying that the output from the agent does not give any obvious ideas as to what to look into? - Vineet
I'm not an expert on the JSON output, but my suspicion based on similar issues is that you're trying to use both dialogflow-fulfillment and actions-on-google libraries and sending a simple response. These two libraries shouldn't be mixed together. There may be an issue with how the simple response object is being represented in the JSON which doesn't match the actions.google.com docs. - Nick Felker
@NickFelker - I am using the dialogflow-fulfillment library - which depends on actions-on-google. Also, the simpleResponse object is not part of the dialogflow-fulfillment library so I cannot use it. - Vineet

2 Answers

0
votes

Looking at the JSON snippet for a simple response in the documentation, you should wrap your item in a simpleResponse element. Additionally, the keys you are using for text and audio responses are incorrect, and should be textToSpeech and displayText.

{
  "payload": {
    "google": {
      "expectUserResponse": true,
      "richResponse": {
        "items": [
          {
            "simpleResponse": {
              "textToSpeech": "Howdy, this is GeekNum. I can tell you fun facts about almost any number, my favorite is 42. What number do you have in mind?",
              "displayText": "Howdy! I can tell you fun facts about almost any number. What do you have in mind?"
            }
          }
        ]
      }
    }
  }
}
0
votes

Inspired by @NickFelker's answer below and researching more into this topic, I was able to get the SSML working by making sure to add the <speak> tags. So this works:

const client = new WebhookClient({ req, res });
let qToSnd = 'Hi <break time=\"500ms\"/> Can I help you?';
let conv = client.conv();
conv.ask('<speak>' + qToSnd + '</speak>');
client.add(conv);

The fulfillment API returns:

{
  "payload": {
    "google": {
      "expectUserResponse": true,
      "richResponse": {
        "items": [
          {
              "simpleResponse": {
                  "textToSpeech": "<speak>Hi <break time=\"500ms\"/> Can I help you</speak>"
              }
          }
        ]
      }
    }
  }
}