duplicates

I am looking for the fastest way to remove duplicate values in a string separated by commas. So my string looks like this; $str = 'one,two,one,five,seven,bag,tea'; I can do it be exploding the string to values and then compare, but I think it will be slow. what about preg_replace() will it be faster? Any one did it using this function? The shortest code would be: $str = implode(',',array_unique(explode(',', $str))); If it is the fastest... I don't know, it is probably faster then looping explicitly. Reference: implode , array_unique , explode Dealing with: $string = 'one,two,one,five,seven,bag

I have approximately 1.7M documents in mongodb (in future 10m+). Some of them represent duplicate entry which I do not want. Structure of document is something like this: { _id: 14124412, nodes: [ 12345, 54321 ], name: "Some beauty" } Document is duplicate if it has at least one node same as another document with same name . What is the fastest way to remove duplicates? Assuming you want to permanently delete docs that contain a duplicate name + nodes entry from the collection, you can add a unique index with the dropDups: true option: db.test.ensureIndex({name: 1, nodes: 1}, {unique: true,

I am writing a Python program to find and remove duplicate files from a folder. I have multiple copies of mp3 files, and some other files. I am using the sh1 algorithm. How can I find these duplicate files and remove them? Fastest algorithm - 100x performance increase compared to the accepted answer (really :)) The approaches in the other solutions are very cool, but they forget about an important property of duplicate files - they have the same file size. Calculating the expensive hash only on files with the same size will save tremendous amount of CPU; performance comparisons at the end,

I have a SQL query where I want to insert multiple rows in single query. so I used something like: $sql = "INSERT INTO beautiful (name, age) VALUES ('Helen', 24), ('Katrina', 21), ('Samia', 22), ('Hui Ling', 25), ('Yumie', 29)"; mysql_query( $sql, $conn ); The problem is when I execute this query, I want to check whether a UNIQUE key (which is not the PRIMARY KEY ), e.g. 'name' above, should be checked and if such a 'name' already exists, the corresponding whole row should be updated otherwise inserted. For instance, in the example below, if 'Katrina' is already present in the database, the

I need to parse a json file which unfortunately for me, does not follow the prototype. I have two issues with the data, but i've already found a workaround for it so i'll just mention it at the end, maybe someone can help there as well. So i need to parse entries like this: "Test":{ "entry":{ "Type":"Something" }, "entry":{ "Type":"Something_Else" } }, ... The json default parser updates the dictionary and therfore uses only the last entry. I HAVE to somehow store the other one as well, and i have no idea how to do this. I also HAVE to store the keys in the several dictionaries in the same

Suppose I have a data.table like this: Table: V1 V2 A B C D C A B A D C I want each row to be regarded as a set, which means that B A and A B are the same. So after the process, I want to get: V1 V2 A B C D C A In order to do that, I have to first sort the table row-by-row and then use unique to remove the duplicates. The sorting process is quite slow if I have millions of rows. So is there an easy way to remove the duplicates without sorting? For just two columns you can use the following trick: dt = data.table(a = letters[1:5], b = letters[5:1]) # a b #1: a e #2: b d #3: c c #4: d b #5: e a