AWS API Gateway Unexpected end of JSON input

Question

We have a company API that we are currently switching over to AWS API Gateway. The endpoints in API Gateway use a Node.js Lambda function to hit our existing internal endpoint, using AWS for rate limiting and authentication. My first endpoint worked perfectly, but my second endpoint is giving me a blank response and in CloudWatch I see the following error:

2017-10-04T03:24:46.957Z 925a40ba-a8b3-11e7-be24-8d954fcaf057 
SyntaxError: Unexpected end of JSON input
at Object.parse (native)
at IncomingMessage.<anonymous> (/var/task/index.js:67:37)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)

If I hit our API directly it properly returns valid JSON

[{"name":"Distinct Zips","offers":8,"affiliates":1,"margin":0,"profit":0,"paid":0,"received":0,"conversion_rate":100,"average_profit":0,"total_calls":1,"qualified_calls":1,"duplicate_calls":0,"returned_calls":0},{"name":"","offers":0,"affiliates":0,"margin":0,"profit":0,"paid":0,"received":0,"conversion_rate":0,"average_profit":0,"total_calls":0,"qualified_calls":0,"duplicate_calls":0,"returned_calls":0},{"name":"Total","offers":8,"affiliates":1,"margin":0,"profit":0,"paid":0,"received":0,"conversion_rate":100,"average_profit":0,"total_calls":1,"qualified_calls":1,"duplicate_calls":0,"returned_calls":0}]

The JSON is valid, so I'm not sure why AWS is returning an error with unexpected end. I tried changing the result to be just a single JSON item, not an array, but still got the same error in CloudWatch. I'm not even sure where to begin looking, if it's likely an issue with our Lambda function, or if it's something with what our codebase is actually returning.

Edits for clarity

The request uses a Lambda function integration but does not use Lambda Proxy integration. The full lambda can be seen at https://gist.github.com/awestover89/a53c0f2811c566c902a473ea22e825a5

We handle chunked data in the Lambda using the callback:

callback = function(response) {
    var responseString = '';

    // Another chunk of data has been recieved, so append it to `str`
    response.on('data', function (chunk) {
        responseString += chunk;
    });

    // The whole response has been received
    response.on('end', function () {
        console.log(responseString);
        // Parse response to json
        var jsonResponse = JSON.parse(responseString);

        var output = {
            status: response.statusCode,
            bodyJson: jsonResponse,
            headers: response.headers
};

How is the APIGateway configured? Under the "Integration Request" section, did you choose "Lambda Function" and "Use Lambda Proxy Integration", so it'll pass everything straight through? — Dan Crews
console.log the event in your handler and then check the CloudWatch logs. Update your question with what you see. — Noel Llevares
It sounds like you are using http/https to fetch a response from your backend, and may have an issue with the way you are capturing the response that causes you to parse only the first chunk of data rather than waiting for the entire response, so the code fails in a way related to the size or timing of chunk arrival of the response. console.log(foo) immediately before you JSON.parse(foo)? i.e., a Lambda issue rather than an actual API Gateway issue. — Michael - sqlbot
We do not use Lambda Proxy Integration, I have updated the question with a gist of the Lambda function. console.log of the response immediately before JSON.parse has just an empty string in CloudWatch. — awestover89
I think I found the problem, the query string isn't properly getting sent over to our direct endpoint and that's causing the result to come back malformed (query string parameters are required) — awestover89

awestover89 awestover89 · Accepted Answer · 2017-10-05T19:48:04

Problem was on our application's end. API Gateway sets the Content-Type header for all requests to our endpoints as application/json, and our application scrubs all query string parameters when the content-type is set to JSON, in favor of pulling the body.

This endpoint had some required query string parameters that were getting removed, so it failed and the error message it sent back wasn't properly formatted JSON, hence the inability for Node to parse it.

AWS API Gateway Unexpected end of JSON input

2 Answers