Mongoose limiting query to 1000 results when I want more/all (migrating from 2.6.5 to 3.1.2)

Question

I\'m migrating my app from Mongoose 2.6.5 to 3.1.2, and I\'m running into some unexpected behavior. Namely I notice that query results are automatically being limited to 100

Answer 1

You're getting tripped up by a pair of related factors:

1. Mongoose's default query batchSize changed to 1000 in 3.1.2.
2. MongoDB has a known issue where a query that requires an in-memory sort puts a hard limit of the query's batch size on the number of documents returned.

So your options are to put a combo index on TestDataPoint that would allow mongo to use it for sorting by dataPoint in this type of query or increase the batch size to at least the total count of documents you're expecting.

Answer 2

Wow that's awful. I'll publish a fix to mongoose soon removing the batchSize default (was helpful when streaming large result sets). Thanks for the pointer.

UPDATE: 3.2.1 and 2.9.1 have been released with the fix (removed batchSize default).