Javascript - Quickly remove duplicates in object array

Question

I have 2 arrays with objects in them such as:

[{"Start": 1, "End": 2}, {"Start": 4, "End": 9}, {"Start": 12, "End": 16}, ... ]

I want to merge the 2 arrays while removing duplicates. Currently, I am doing the following:

array1.concat(array2);

Then I am doing a nested $.each loop, but as my arrays get larger and larger, this takes O(n^2) time to execute and is not scalable.

I presume there is a quicker way to do this, however, all of the examples I have found are working with strings or integers.

Any recommended algorithms or methods out there to make this faster?

Answer 1

This answer bases on the assumption that the order does not matter and you can create unique keys from your objects.

You copy all n entries from the array a to an object c, creating a unique key, after that you copy all m entries from array b to that object (this will automatically eliminate the duplicates) and you are finished in O(n+m):

var a = [{"Start": 1, "End": 2}, {"Start": 4, "End": 9}];
var b = [{"Start": 4, "End": 9}, {"Start": 3, "End": 12}];

var c = {};
a.forEach(function(el){c[el.Start+".."+el.End] = el});
b.forEach(function(el){c[el.Start+".."+el.End] = el});

console.log(c);
// yields: {"1..2":{"Start": 1, "End": 2},"4..9":{"Start": 4, "End": 9},"3..12":{"Start": 3, "End": 12}}

This notation in this object is a bit redundant but you are extremely fast on merging the two arrays. Maybe this could be improved further.

Answer 2

Sort the objects first, low to high. O(n log n) with quicksort.

Then you can make pruning algorithm that can take advantage of this sorting to loop through both arrays in one loop in O(2n).

The merge the original array and the pruned array.

---

Keep in mind though that objects in JavaScript have no order, you can't sort them. Convert to an array, keep references and sort that.

Answer 3

I'm not as familiar with javascript, so not 100% sure this is doable (not sure about subtlties of comparing objects for equality, etc), but in java or other languages, something like this might work:

• Iterate over first array.
• For each element store into a "counter" hashmap where key is the object and value is the count.

After this first pass, you should have something like:

{{"Start": 1, "End": 2}:1, {"Start": 4, "End": 9}:1, {"Start": 12, "End": 16}:1, ... }

• Then, iterate over the second array
• For each element, lookup current element inside counter hashmap.
• If the counter hashmap contains a key that matches the current element it's a duplicate
• Otherwise, append to the first array.

Might be a tad faster than having to sort first (if it's possible to use objects as keys, that is)?