Using Amazon SQS with multiple consumers

Question

I have a service-based application that uses Amazon SQS with multiple queues and multiple consumers. I am doing this so that I can implement an event-based architecture and

Answer 1

I think you are doing it wrong.

It looks to me like you are using the same queue to do multiple different things. You are better of using a single queue for a single purpose.

Instead of putting an event into the 'registration-new' queue and then having two different services poll that queue, and BOTH needing to read that message and both doing something different with it (and then needing a 3rd process that is supposed to delete that message after the other 2 have processed it).

One queue should be used for one purpose.

• Create a 'index-user-search' queue and a 'send to mixpanels' queue, so the search service reads from the search queues, indexes the user and immediately deletes the message.

• The mixpanel-service reads from the mix-panels queue, processes the
  message and deletes the message.

The registration service, instead of emiting a 'registration-new' to a single queue, now emits it to two queues.

To take it one step better, add SNS into the mix here and have the registration service emit an SNS message to the 'registration-new' topic (not queue), and then subscribe both of the queues I mentioned above, to that topic in a 'fan-out' pattern.

https://aws.amazon.com/blogs/aws/queues-and-notifications-now-best-friends/

Both queues will receive the message, but you only load it into SNS once - if down the road a 3rd unrelated service needs to also process 'registration-new' events, you create another queue and subscribe it to the topic as well - it can run with no dependencies or knowledge of what the other services are doing - that is the goal.