AWS Lambda - Nodejs: Allocation failed - JavaScript heap out of memory

Question

I am using Lambda (nodeJS) to query noSQL data (dynamo db).

Let's say I have table "student" in DynamoDB, and I have an API which return list of students for a specific class (class_id). (I used "query" )

As I know, the dynamo paginates the result, so my API works like below:

{class_id : 'xxxx'} => return 1st list of students
{class_id : 'xxxx', last_evaluated_key: { ....} => return next list of student (if LastEvaluatedKey exists)

My lambda code :

exports.handler = function(e, ctx, callback) {
    var rp = require('request-promise');

    var students = [];

    var hasMore = true;
    var params = {
        class_id: e.class_id
    }

    while (hasMore) {
        var options = {
            method: 'POST',
            uri: 'https://xxxxxx.eu-west-1.amazonaws.com/dynamodliblightdm-gobject-1-0:amd64liblightdm-gobject-1-0:amd64b/getStudents',
            body: params,
            json: true // Automatically stringifies the body to JSON
        };

        rp(options)
            .then(function(repos) {
                console.log('count: ' + repos.Count);
                students.push(repos.Items);

                if (repos.hasOwnProperty("LastEvaluatedKey")) {
                    params['last_evaluated_key'] = repos.LastEvaluatedKey;
                } else {
                    hasMore = false;
                }

            })
            .catch(function(err) {
                console.log('Error', err);
            });
    }


    callback(null, 'done.');
}

I got error:

42676 ms: Mark-sweep 804.1 (954.3) -> 802.7 (954.3) MB, 1803.0 / 0.0 ms (+ 246.3 ms in 32 steps since start of marking, biggest step 35.7 ms) [allocation failure] [GC in old space requested]. 44415 ms: Mark-sweep 802.7 (954.3) -> 802.7 (954.3) MB, 1738.6 / 0.0 ms [allocation failure] [GC in old space requested]. 46318 ms: Mark-sweep 802.7 (954.3) -> 809.5 (859.3) MB, 1902.9 / 0.0 ms [last resort gc]. 48184 ms: Mark-sweep 809.5 (859.3) -> 816.4 (858.3) MB, 1865.7 / 0.0 ms [last resort gc]. <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x322e8723fa99 2: new constructor(aka Multipart) [/var/task/lambda-func/node_modules/request/lib/multipart.js:~8] [pc=0x1b47df3f5f98] (this=0x1175e583149 ,request=0x1175e582fa9 ) 4: new constructor(aka Request) [/var/task/lambda-func/node_modules/request/request.js:125] [pc=0x1b47df4df3e6] (this=0x1175e... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/var/lang/bin/node] 2: 0x55d79ff0b302 [/var/lang/bin/node] 3: v8::Utils::ReportApiFailure(char const*, char const*) [/var/lang/bin/node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/var/lang/bin/node] 5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/var/lang/bin/node] 6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/var/lang/bin/node] 7: 0x1b47df2062bf

Any suggestion is appreciated.

That looks like in Infinite loop. The while loop is not waiting until your promise is resolved. It just calls the same thing over and over again — yBrodsky
Hi @yBrodsky, I can see the problem came from the while loop, my question is how to use 'request-promise' in while loop, or a bigger question is how to get all items from dynamodb properly. — Phong Vu

yBrodsky yBrodsky · Accepted Answer · 2018-04-19T13:32:45

Suggestion to get all the students.

const fetch  = (lastEvaluatedKey)  => {
  return rp().then((res) => {
    students = students.concat(res.moreStudents);
    if(res.shouldKeepFetching) {
      return fetch(res.lastKey);
    }

    return Promise.resolve();
  })
}

fetch().then(() => {
  //you fetched them all
})

If there are a shitload of students this might get you with another out of memory problem. Another thing you can do, but I don't know if lambda allows it yet, is to use a for loop with async/await

AWS Lambda - Nodejs: Allocation failed - JavaScript heap out of memory

1 Answers