How to deduplicate documents while indexing into elasticsearch from logstash

Question

I\'m using Logstash 1.4.1 together with ES1.01 and would like to replace already indexed documents based on a calculated checksum. I\'m currently using the \"fingerprint\" f

Answer 1

Assuming the fingerprint is getting set as the _id, you may be hitting an issue with logstash's daily index management and not using the timestamp from your data.

Ensure that you have your timestamp set from the input data, so you are guaranteed the document goes to the correct daily index:

http://logstash.net/docs/1.4.2/filters/date

If my guess is correct, you should see that your duplicate documents have different @timestamp and are in different daily indexes.