How to wait for async actions inside AWS Lambda?

Question

I am trying to process uploaded file in S3. Since getObject is asyncronous main function ends before processing is done, and AWS kills lambda in 3-4 seconds

Answer 1

Using async/await

let AWS = require('aws-sdk');
let lambda = new AWS.Lambda();
let data;
exports.handler = async (event) => {
    try {
        data = await lambda.getAccountSettings().promise();
    }
    catch (err) {
        console.log(err);
        return err;
    }
    return data;
};

Answer 2

async is not included but that does not mean you cannot add it yourself. Simply add the package locally (npm install async), and include the node_modules folder in your ZIP before uploading your Lambda function.

If you want to handle dev dependencies separately (e.g.: test, aws-sdk to execute your function locally, etc), you can add them under devDependencies in your package.json. Also, if you want to automate the process of developing, testing, deploying and promoting your code, these two repos will turn out to be very handy.

Grunt routine to test, package and deploy your lambdas

Command line tool for running and deploying your lambda functions

Answer 3

The life of a dev is constantly changing and we now have NodeJS 8 on lambda. For anyone looking at this now check out:

Lambda node 8.10 vs node 6.10 comparison: https://aws.amazon.com/blogs/compute/node-js-8-10-runtime-now-available-in-aws-lambda/

Basics of JS async: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

Even more aws sdk examples: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html

Details on wtf the .promise() method is in the first link: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html#promise-property

Here is my take at a basic example (try pasting into your own lambda):

exports.handler = async (event) => {    
    function wait(){
        return new Promise((resolve, reject) => {
            setTimeout(() => resolve("hello"), 2000)
        });
    }
    
    console.log(await wait());
    console.log(await wait());
    console.log(await wait());
    console.log(await wait());
    console.log(await wait());
    console.log(await wait());
    
    return 'exiting'
};

The above yields:

As you can see it waited 12 seconds without killing my function :)

TODO more than one thing per await, use Promise.all([]) syntax like this:

exports.handler = async (event) => {
    var uploadPromises = [];
    folder.files.forEach(file => {
        uploadPromises.push( s3.putObject({
            Bucket: "mybucket",
            Key: file.name,
            Body: file.data
        }).promise());
    });

    await Promise.all(uploadPromises);
    return 'exiting'
}; 

Orignal Answer Below

I had the exact same issue on my hands.

The problem is the javascript event loop is empty so Lambda thinks it's done.

This is how I solved this problem. I realize this is not ideal, and I wish there was a better way, but I didn't want to a) add libraries, b) coordinate lambda invocations, or c) switch to another language.

At the end of the day it works.

    exports.handler = (event, context, callback) => {
        var response;
        var callBackCount;

        /*
        Ensures the javascript event loop is never empty.
        This is the key to keeping lambda from exiting early
        */
        setInterval(function(){}, 1000);

        /*
        Tell lambda to stop when I issue the callback.
        This is super important or the lambda funciton will always go until it hits the timeout limit you set.
        */
        context.callbackWaitsForEmptyEventLoop = false;
        
        //My way of determining when I'm done with all calls
        callBackCount = 0;
      
        //My info to return
        response = "";
        
        //Various functions that make rest calls and wait for a response
        asyncFunction1();
        asyncFunction2();
        asyncFunction3();

        //Same for asyncFunction 2 and 3
        function asyncFunction1(){
          response += callBackResponseForThisMethod;
      
          returnResponse();
        }

        function returnReponse(){
            callBackCount++;

            if(callBackCount == 3){
              //Lambda will stop after this as long as    context.callbackWaitsForEmptyEventLoop was set to false 
              callback(null, JSON.stringify(response));
            }
        }

    };

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html

Answer 4

You might want to make a synchronous call instead; since you seem to be processing your file in the same lambda function.

If for some reason you want to get a callback; you can do that by either directly invoking lambda or via something that would produce a lambda event. Note that lambda functions are supposed be stateless; so you should pass in all of the needed information.

Answer 5

If you want to use require('async'); pack or require('sleep'); pack as well you need to upload your function as a zip file like this:

Creating a Deployment Package (Node.js)

Zip all the content of the folder as I explain in this question as well:

MQTT in AWS Lambda function for Alexa Javascript

About the synchronous processing, you can use require('async'); normally, is just to use the async.series function like this:

    async.series([
    function(callback) {
        // to do the function 1
        callback();
    },
    function(callback) {
        // to do the function 2
        callback();
    },
    function(callback) {
        // to do the function 3
        callback();
    }
], function(err) {
    // to do the function if any error happens...

    if (err) {
        //...
    }
    //....
});

This way the lambda function will works synchronously.

I hope help you.

Answer 6

I think your lambda function should end with a context.done() call. For instance, try adding it this way:

s3.getObject(params, function(err, data) {
    if (err) {
         ...
        context.done("Error: " + err.stack);
    } else {
        processFile(data.Body.toString(), 0);
        console.log("ok");
        context.done(null, "success");
    }
});