How to prevent duplicate SQS Messages?

Question

What is the best way to prevent duplicate messages in Amazon SQS? I have a SQS of domains waiting to be crawled. before I add a new domain to the SQS I can check with the saved

Answer 1

According to AWS Docs, Exactly-Once Processing provides a way to avoid duplicate messages.

Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.

If your queue is an FIFO queue and has enabled content based duplication, this function can be utilized to avoid duplicate messages during the deduplication interval. For more, read this section, and the below link.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html#cfn-sqs-queue-contentbaseddeduplication