Firestore Deadline Exceeded Node

问题

I would like to load collection that is ~30k records. I.e load it via.

const db = admin.firestore();
let documentsArray: Array<{}> = [];
db.collection(collection)
  .get()
  .then(snap => {
    snap.forEach(doc => {
      documentsArray.push(doc);
    });
  })
  .catch(err => console.log(err));

This will always throw Deadline Exceeded error. I have searched for some sorts of mechanism that will allow me to paginate trough it but I find it unbelievable not to be able to query for not that big amount in one go.

I was thinking that it may be that due to my rather slow machine I was hitting the limit but then I deployed simple express app that would do the fetching to app engine and still had no luck.

Alternatively I could also export the collection with gcloud beta firestore export but it does not provide JSON data.

回答1:

I'm not sure about firestore, but on datastore i was never able to fetch that much data in one shot, I'd always have fetch pages of about 1000 records at a time and build it up in memory before processing it. You said:

I have searched for some sorts of mechanism that will allow me to paginate trough

Perhaps you missed this page

https://cloud.google.com/firestore/docs/query-data/query-cursors

回答2:

In the end the issue was that machine that was processing the 30k records from the Firestore was not powerful enough to get the data needed in time. Solved by using, GCE with n1-standard-4 GCE.

来源：https://stackoverflow.com/questions/55117181/firestore-deadline-exceeded-node