Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: [Chinese letters]

Question

I'm building my own WebhookClient for dialog flow. My code is the following (using Azure Functions, similar to Firebase Functions):

module.exports = async function(context, req) {
    const agent = new WebhookClient({ request: context.req, response: context.res });

    function welcome(agent) {
        agent.add(`Welcome to my agent!!`);
    }

    let intentMap = new Map();

    intentMap.set("Look up person", welcome);

    agent.handleRequest(intentMap);
}

I tested the query and the response payload looks like this:

{
    "fulfillmentText": "Welcome to my agent!!",
    "outputContexts": []
}

And the headers in the response look like this:

Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Tue, 11 Dec 2018 18:16:06 GMT

But when I test my bot in dialog flow, it returns the following:

Webhook call failed. Error: Failed to parse webhook JSON response: Expect message object but got: "笀ഀ਀  ∀昀甀氀昀椀氀氀洀攀渀琀吀攀砀琀∀㨀 ∀圀攀氀挀漀洀攀 琀漀 洀礀 愀最攀渀琀℀℀∀Ⰰഀ਀  ∀漀甀琀瀀甀琀䌀漀渀琀攀砀琀猀∀㨀 嬀崀ഀ਀紀".

There's Chinese symbols!? Here's a video of me testing it out in DialogFlow: https://imgur.com/yzcj0Kw

Answer 1

I know this should be a comment (as it isn't really an answer), but it's fairly verbose and I didn't want it to get lost in the noise.

I have the same problem using WebAPI on a local machine (using ngrok to tunnel back to Kestrel). A friend of mine has working code (he's hosting in AWS rather than Azure), so I started examining the differences between our responses. I've notice the following:

1. This occurs with Azure Functions and WebAPI (so it's not that)
2. The JSON payloads are identical (so it's not that)
3. Working payload isn't chunked
4. Working payload doesn't have a content type

As an experiment, I added this code to Startup.cs, in the Configure method:

app.Use(async (context, next) =>
{
    var original = context.Response.Body;
    var memory = new MemoryStream();

    context.Response.Body = memory;
    await next();

    memory.Seek(0, SeekOrigin.Begin);

    if (!context.Response.Headers.ContentLength.HasValue)
    {
        context.Response.Headers.ContentLength = memory.Length;
        context.Response.ContentType = null;
    }

    await memory.CopyToAsync(original);
});

This code disables response chunking, which is now causing a new and slightly more interesting error for me in the google console:

*Webhook call failed. Error: Failed to parse webhook JSON response: com.google.gson.stream.MalformedJsonException: Unterminated object at line 1 column 94 path $.\u0000\\"\u0000f\u0000u\u0000l\u0000f\u0000i\u0000l\u0000l\u0000m\u0000e\u0000n\u0000t\u0000M\u0000e\u0000s\u0000s\u0000a\u0000g\u0000e\u0000s\u0000\\"\u0000.\

I thought this could be encoding at first, so I stashed my JSON as a string and used the various Encoding classes to convert between them, to no avail.

I fired up Postman and called my endpoint (using the same payload as Google) and I can see the whole response payload correctly - it's almost as if Google's end is terminating the stream part-way through reading...

Hopefully, this additional information will help us figure out what's going on!

Update

After some more digging and various server/lambda configs, I spotted this post here: https://github.com/googleapis/google-cloud-dotnet/issues/2258

It turns out that json.net IS the culprit! I guess it's something to do with the formatters on the way out of the pipeline. In order to prove this, I added this hard-coded response to my POST controller and it worked! :)

return new ContentResult()
{
    Content = "{\"fulfillmentText\": null,\"fulfillmentMessages\": [],\"source\": null,\"payload\": {\"google\": {\"expectUserResponse\": false,\"userStorage\": null,\"richResponse\": {\"items\": [{\"simpleResponse\": {\"textToSpeech\": \"Why hello there\",\"ssml\": null,\"displayText\": \"Why hello there\"}}],\"suggestions\": null,\"linkOutSuggestion\": null}}}}",
    ContentType = "application/json",
    StatusCode = 200
};

Answer 2

Despite the HTTP header saying the charset is utf-8, that is definitely using the utf-16le character set, and then the receiving side is treating them as utf-16be. Given you're running on Azure, it sounds like there is some configuration you need to make in Azure Functions to represent the output as UTF-8 instead of using UTF-16 strings.