Closing DAX Client in Lambda Function

Question

I have a node.js functon that I want to deploy as a AWS Lambda function. I have noticed that when I use redis Elasticache, I must close the connection I opened with redis.createClient or else the Lambda function times out. I do this by simply calling the client's quit() method. If I do this prior to issuing the Lambda callback, the Lambda function ends as expected. If I do not, the Lambda function times out at whatever timeout interval I set. In a test setting, I have a Lambda function that operated on the cache and terminates in 30 milliseconds. Without the call to the redis client quit() method, the same Lambda function won't terminate before a 1 minute timeout. I'm not sure, but I assume the Lambda function behaves this way because it doesn't know whether the function is done executing (even after the callback) since the redis connection is still active.

I'm not all that concerned about this because it's easy enough to call the quit() method. The problem I am having is when I try to do something similar with a DynamoDB DAX client. I can have the Lambda function access DynamoDB directly through my VPC endpoint and everything works fine. If I use the DAX client, the modifications to the table actually take place (I inserted an item and can see that it is there), but the Lambda function always times out.

Here is some sample code:

const AmazonDaxClient = require('amazon-dax-client');
const AWS             = require('aws-sdk');
const config          = require('./config');

AWS.config.update(config.aws);
var ddbClient = new AWS.DynamoDB.DocumentClient(config.dynamodb);
var dax = new AmazonDaxClient(config.dax);
var daxClient = new AWS.DynamoDB.DocumentClient({service: dax });

If I just use the ddbClient, everything works. If I use the daxClient, everything also works (inserts, deletes, updates, etc.), but the Lambda function times out. Is there a similar quit() method for the daxClient that tells the DAX cluster I'm done and it can close all the connections cleanly? I suspect the problem with this is that they want the daxClient to behave exactly like a normal ddbClient, and there is not a quit() method for the ddbClient.

Answer 1

It looks like the Lambda context property callbackWaitsForEmptyEventLoop, as documented here, may be relevant to your case. By default it is set to true, which means that even if your handler returns a value via callback, as long as there is anything still in the Node event loop, the Lambda won't complete its execution.

Setting it to false will allow your Lambda handler to complete even with things in the event loop, but be aware that these logical threads will freeze execution while the Lambda is not running. This behavior might not be appropriate for all libraries, depending on how robust they are.

On an interesting side note, the old Lambda return style of calling context.succeed (rather than callback) will not wait for an empty event loop, regardless of the property's setting. (This gist goes into some extra detail.)

Answer 2

As of version 1.1.2 of the DAX JS client, this is no longer an issue - the Lambda event loop will pause correctly when using DAX.