MongoDB's performance on aggregation queries

Question

After hearing so many good things about MongoDB\'s performance we decided to give Mongodb a try to solve a problem we have. I started by moving all the records we have in se

Answer 1

Couple things.

1) Your group query is processing lots of data. While your result set is small, it looks like it's doing a table scale of all of the data in your collection in order to generate that small result. This is probably the root cause of the slowness. To speed this up, you might want to look at the disk performance of your server through iostat while the query is running as that is likely the bottleneck.

2) As has been pointed out in other answers, the group command uses the javascript interpreter, which is going to limit performance. You might try using the new aggregation framework that is released as beta in 2.1 (note: this is an unstable release as of Feb 24 2012). See http://blog.mongodb.org/post/16015854270/operations-in-the-new-aggregation-framework for a good introduction. This won't overcome data volume problem in (1), but it is implemented in C++ and if javascript time is the bottleneck, then it should be much faster.

3) Another approach would be to use incremental map-reduce to generate a second collection with your grouped results. The idea is that you'd run a map-reduce job to aggregate your results once, and then periodically run another map-reduce job that re-reduces new data into the existing collection. Then you can query this second collection from your app rather than running a group command every time.