问题
I have a push task queue and each of my jobs consists of multiple similar TaskQueue tasks. Each of these tasks takes less than a second to finish and can add new tasks to the queue (they should be also completed to consider the job finished). Task results are written to a DataStore.
The goal is to understand when a job has finished, i.e. all of its tasks are completed.
Writes are really frequent and I can't store the results inside one entity group. Is there a good workaround for this?
回答1:
In a similar context I used a scheme based on memcache, which doesn't have a significant write rate limitation as datastore entity groups:
- each job gets a unique memcache key associated with it, which it passes to each of subsequent execution tasks it may enqueue
- every execution task updates the memcache value corresponding to the job key with the current timestamp and also enqueues a completion check task, delayed with an idle timeout value, large enough to declare the job completed if elapsed.
- every completion check task compares the memcache value corresponding to the job key against the current timestamp:
- if the delta is less than the idle timeout it means the job is not complete (some other task was executed since this completion check task was enqueued, hence some other completion check task is in the queue)
- otherwise the job is completed
Note: the idle timeout should be larger than the maximum time a task might spend in the queue.
来源：https://stackoverflow.com/questions/44934713/figure-out-group-of-tasks-completion-time-using-taskqueue-and-datastore