Mongo error when using aggregation: sort exceeded memory limit

问题

I get the mongo error exceeded memory limit with error code 16819 when I use aggregation sort.

Im using mongo 2.6.

The query is as follows:

db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.aggregate([
     { "$project" : { "visitor_localdate" : 1 , "_id" : 0}}, 
     { "$sort" : { "visitor_localdate" : -1}}
])

回答1:

By default aggregation in MongoDB occurs in memory and pipeline stages have limit of 100 Mb RAM. Looks like you have exceeded this threshold. To handle large dataset you should enable aggregation pipeline stages to write data to temporary files. Use allowDiskUse option for that:

db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.aggregate([
    { "$project" : { "visitor_localdate" : 1 , "_id" : 0}},
    { "$sort" : { "visitor_localdate" : -1}}
], { "allowDiskUse" : true })

回答2:

Use { allowDiskUse: true } just after aggregation pipeline, like below:

db.collectionOrView.aggregate([], { allowDiskUse: true });

回答3:

In case you are using aggregate queries.Put an index on the field by which you are sorting and then use sort operator.

Note: Place the sort operator at the beginning of the pipeline or before the $project, $unwind, and $group aggregation operators. If $project, $unwind, or $group occur prior to the $sort operation, $sort cannot use any indexes. [https://docs.mongodb.com/manual/reference/operator/aggregation/sort/][1]

回答4:

You don't need aggregation for this at all. Use the query

db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.find({}, { "_id" : 0, "visitor_localdate" : 1 }).sort({ "visitor_localdate" : -1 })

and put an index on visitor_localdate. This is simpler and faster than aggregation.

来源：https://stackoverflow.com/questions/26375017/mongo-error-when-using-aggregation-sort-exceeded-memory-limit

标签

mongodb

aggregation-framework