Dialogflow fails to parse the JSON response from my webhook seems to change character encoding

Question

I have a fulfillment webhook which I am trying to use to return a simple response to a test intent on Dialogflow.

The webhook is an azure function and having tested it with postman I can see the JSON response being formatted correctly with the right headers etc.

This is the raw response which Dialogflow shows me is the error (Scroll right to see the error on the message)

{
  "responseId": "6d2686c1-a87b-4f64-ac4d-f3a5e706763c",
  "queryResult": {
    "queryText": "Test",
    "parameters": {},
    "allRequiredParamsPresent": true,
    "fulfillmentMessages": [
      {
        "text": {
          "text": [
            ""
          ]
        }
      }
    ],
    "intent": {
      "name": "projects/mytestappname-c0f76/agent/intents/3d70ef1d-9723-451f-b79d-b40f4e16d8b0",
      "displayName": "Test"
    },
    "intentDetectionConfidence": 1,
    "diagnosticInfo": {
      "webhook_latency_ms": 600
    },
    "languageCode": "en"
  },
  "webhookStatus": {
    "code": 3,
    "message": "Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: \"笀∀昀漀氀氀漀眀甀瀀䔀瘀攀渀琀䤀渀瀀甀琀∀㨀渀甀氀氀Ⰰ∀昀甀氀昀椀氀氀洀攀渀琀䴀攀猀猀愀最攀猀∀㨀嬀笀∀戀愀猀椀挀䌀愀爀搀∀㨀渀甀氀氀Ⰰ∀挀愀爀搀∀㨀渀甀氀氀Ⰰ∀挀愀爀漀甀猀攀氀匀攀氀攀挀琀∀㨀渀甀氀氀Ⰰ∀椀洀愀最攀∀㨀渀甀氀氀Ⰰ∀氀椀渀欀伀甀琀匀甀最最攀猀琀椀漀渀∀㨀渀甀氀氀Ⰰ∀氀椀猀琀匀攀氀攀挀琀∀㨀渀甀氀氀Ⰰ∀瀀愀礀氀漀愀搀∀㨀渀甀氀氀Ⰰ∀瀀氀愀琀昀漀爀洀∀㨀渀甀氀氀Ⰰ∀焀甀椀挀欀刀攀瀀氀椀攀猀∀㨀渀甀氀氀Ⰰ∀猀椀洀瀀氀攀刀攀猀瀀漀渀猀攀猀∀㨀渀甀氀氀Ⰰ∀猀甀最最攀猀琀椀漀渀猀∀㨀渀甀氀氀Ⰰ∀琀攀砀琀∀㨀笀∀琀攀砀琀∀㨀嬀∀吀栀椀猀 椀猀 愀 昀甀氀昀椀氀氀洀攀渀琀 洀攀猀猀愀最攀∀崀Ⰰ∀䔀吀愀最∀㨀渀甀氀氀紀Ⰰ∀䔀吀愀最∀㨀渀甀氀氀紀崀Ⰰ∀昀甀氀昀椀氀氀洀攀渀琀吀攀砀琀∀㨀∀吀栀椀猀 椀猀 昀甀氀昀椀氀氀洀攀渀琀 琀攀砀琀∀Ⰰ∀漀甀琀瀀甀琀䌀漀渀琀攀砀琀猀∀㨀渀甀氀氀Ⰰ∀瀀愀礀氀漀愀搀∀㨀笀∀最漀漀最氀攀∀㨀笀∀攀砀瀀攀挀琀唀猀攀爀刀攀猀瀀漀渀猀攀∀㨀琀爀甀攀Ⰰ∀爀椀挀栀刀攀猀瀀漀渀猀攀∀㨀笀∀椀琀攀洀猀∀㨀嬀笀∀搀椀猀瀀氀愀礀吀攀砀琀∀㨀渀甀氀氀Ⰰ∀猀猀洀氀∀㨀渀甀氀氀Ⰰ∀琀攀砀琀吀漀匀瀀攀攀挀栀∀㨀∀吀栀椀猀 椀猀 䐀愀瘀攀✀猀 吀攀猀琀椀渀最 吀攀砀琀∀Ⰰ∀䔀吀愀最∀㨀渀甀氀氀紀崀紀Ⰰ∀昀椀渀愀氀刀攀猀瀀漀渀猀攀∀㨀笀∀椀琀攀洀猀∀㨀嬀笀∀搀椀猀瀀氀愀礀吀攀砀琀∀㨀渀甀氀氀Ⰰ∀猀猀洀氀∀㨀渀甀氀氀Ⰰ∀琀攀砀琀吀漀匀瀀攀攀挀栀∀㨀∀吀栀椀猀 椀猀 䐀愀瘀攀✀猀 䘀椀渀愀氀 吀攀猀琀椀渀最 吀攀砀琀∀Ⰰ∀䔀吀愀最∀㨀渀甀氀氀紀崀紀紀紀Ⰰ∀猀漀甀爀挀攀∀㨀∀攀砀愀洀瀀氀攀⸀挀漀洀∀Ⰰ∀䔀吀愀最∀㨀渀甀氀氀紀\"."
  }
}

However in the same window it shows me what my actual fullfilment response was which is this

{
  "followupEventInput": null,
  "fulfillmentMessages": [
    {
      "basicCard": null,
      "card": null,
      "carouselSelect": null,
      "image": null,
      "linkOutSuggestion": null,
      "listSelect": null,
      "payload": null,
      "platform": null,
      "quickReplies": null,
      "simpleResponses": null,
      "suggestions": null,
      "text": {
        "text": [
          "This is a fulfillment message"
        ],
        "ETag": null
      },
      "ETag": null
    }
  ],
  "fulfillmentText": "This is fulfillment text",
  "outputContexts": null,
  "payload": {
    "google": {
      "expectUserResponse": true,
      "richResponse": {
        "items": [
          {
            "displayText": null,
            "ssml": null,
            "textToSpeech": "This is Dave's Testing Text",
            "ETag": null
          }
        ]
      },
       "finalResponse": {
        "items": [
          {
             "displayText": null,
             "ssml": null,
            "textToSpeech": "This is Dave's Final Testing Text",
            "ETag": null
           }
        ]
      }
    }
  },
  "source": "example.com",
  "ETag": null
}

These are the headers on the response

"Content-Type": "application/json; charset=utf-8", "Date": "Wed, 30 May 2018 07:58:36 GMT", "Server": "Kestrel", "Transfer-Encoding": "chunked",

I've put the response through a validator and it's definitely valid JSON.

The issue isn't the content of the response JSON, I can fix that later if it's incorrect, the issue is that it can't even begin to parse it.

Does anyone have any idea why Dialogflow can't read it at all?

Answer 1

I was getting the same error. DialogFlow was not able to parse the JSON response, displaying some weird Chinese characters. And like you said, the fulfillment response tab would display the message I was sending back!

The only difference is I'm not using Azure functions. My webhook is a .NET Core 2 Web Api and I was returning an OkObjectResult which was definitely producing a valid JSON response.

I haven't figured out why but I managed to get DialogFlow to parse the message by not returning an object but with the following:

HttpContext.Response.WriteAsync("{ \"fulfillmentText\" : \"This is a message from my webhook !\" }");

Answer 2

The root problem is actually ASP.NET Core by default uses transfer-encoding: chunked for ActionResult and for some reason Dialogflow does not support parsing chunked transfer (most clients should handle it transparently). You can verify this by looking at the raw server response using a network debugger.

Unfortunately there doesn't seem to be an easy way to disable chunked responses in ASP.NET Core 2 (there was in Core 1).

The workaround is to use a ContentResult response type which seems to only return buffered responses so it will not use chunked transfer. But obviously you'll need to handle your own JSON serialization so it should look something like

    [HttpGet]
    public ContentResult Test()
    {
        var response = new()
        {
            Hello = "World"
        };

        return Content(JsonConvert.SerializeObject(response), "application/json");
    }