How to prevent duplicate SQS Messages?

Question

What is the best way to prevent duplicate messages in Amazon SQS? I have a SQS of domains waiting to be crawled. before I add a new domain to the SQS I can check with the saved

Answer 1

There is no API level way of preventing duplicate messages to be posted to a SQS queue. You would need to handle this at application level I am afraid.

You can use a DynamoDB table to store your Domain Names waiting to be crawled and only add them to the queue if they are not in DynamoDB for example.