How to get all messages in Amazon SQS queue using boto library in Python?

Question

I\'m working on an application whose workflow is managed by passing messages in SQS, using boto.

My SQS queue is growing gradually, and I have no way to check how many e

Answer 1

My understanding is that the distributed nature of the SQS service pretty much makes your design unworkable. Every time you call get_messages you are talking to a different set of servers, which will have some but not all of your messages. Thus it is not possible to 'check in from time to time' to set if a particular group of messages are ready, and then just accept those.

What you need to do is poll continuously, take all the messages as they arrive, and store them locally in your own data structures. After each successful fetch you can check your data structures to see if a complete set of message has been collected.

Keep in mind that messages will arrive out of order, and some messages will be delivered twice, as deletes have to propagate to all the SQS servers, but subsequent get requests sometimes beat out the delete messages.