task-queue

问题 I need to build "tasks scheduler" with Amazon tools. Main problem that i need execute task once with huge delay (it may be a few hours or few weeks). I try to research how to build it with CloudWatch and Lambda function. As i understand - i need to use separate rule for one timeout execution. But AWS allow me only 100 rules/account. Probably i'm going wrong way and this tools not intended for my task. Also i tried SQS, but it did't allow me to set timeout more than 15 minutes. The simplest

问题 I'm using Google App Engine and I've been using a named push queue to process some tasks. The queue was defined and configured using queue.xml. That's fine. Now I'm trying to do more: 1) Is there a way to dynamically create and configure a new queue in Java "on the fly"? 2) Also, once I have a queue (either through queue.xml or 1) if possible), is there a way to pause and resume the queue in Java (just like you can do using the admin console buttons)? Thanks! 回答1: 1) since it has to be in the

问题 I am looking for a WorkQueue API offering the following features: java.util.Queue compatible offers (optional) Set-semantic single- and batch-processing concurrency (of course) scheduling different processing policies wait until next scheduled execution pre-process, if batch size met delayed processing (minimum time in queue, before being processed) persistence (optional) There are a lot of interesting implementations in the jdk, e.g. java.util.DelayQueue which i could use. I just wanted to

问题 I had converted some tasks to run on a dynamic backend. The tasks are failing silently [no logged error, no retry, nothing] ~20% of the time (min:10%, max:60%, sample:large, long term). Switching the task away from the backend restores retries and gets the failure rate back to ~0%. Any ideas? 回答1: Converting it to a backend exacerbated the problem but wasn't the problem. I had specified a task_retry_limit and the queue was a push queue. With a backend the number of instances is specified. (I

问题 The MPI implementation I am working with does not natively support full multi-thread operations (highest level is MPI_THREAD_SERIALIZED , for complicated reasons), so I am trying to funnel the requests from multiple threads into a single worker thread and then scattering the results back out to multiple threads. I can handle gathering local request tasks easily enough by using a concurrent queue, and MPI natively supports enqueuing asynchronous tasks. However, the problem is getting the two

问题 I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs. I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so jobs enqueued. What are good alternatives to this module which can be used in Perl? 回答1: I've had pretty good success with using ZeroMQ for this sort of problem, both with Perl and other languages. In my experience, the ZeroMQ module appears to be

问题 I have written a servlet code in Java for reading a line from file which is stored in Google Cloud Storage . Once I read each line I pass it to prediction API . Once i get the prediction of the text passed . I append it to original line and store it in some other file in Google cloud storage . This sources file is a csv and has more than 10,000 records . Since I am parsing it individually,passing it to prediction API and then storing back to Cloud Storage . It takes lot of time to do so .

问题 I'm looking at Google App Engine's new task queue API for Java and I'm having a hard time relating that to my GWT application. If I wanted to use a task queue to do some asynchronous processing, how should I do that using GWT. The way I see it is, I'd have to send a server request that would then do the submission to the task queue API. If I understand task queues properly, I'd have to create yet another servlet to do the processing from the task queue (be the worker). I'm looking for 2

问题 I have a simple Google App Engine app that includes a /update page which updates a YouTube playlist. It looks like this: class UpdatePage(webapp2.RequestHandler): @decorator.oauth_required def get(self): update_result = self.update_playlist() ... routes = [('/update', UpdatePage), (decorator.callback_path, decorator.callback_handler())] app = webapp2.WSGIApplication(routes, debug=True) It works as expected and the update_playlist() method does its job, but it turns out that under some

问题 I'm using testbed to unit test my google app engine app, and my app uses a taskqueue. When I submit a task to a taskqueue during a unit test, it appears that the task is in the queue, but the task does not execute. How do I get the task to execute during a unit test? 回答1: The dev app server is single-threaded, so it can't run tasks in the background while the foreground thread is running the tests. I modified TaskQueueTestCase in taskqueue.py in gaetestbed to add the following function: def