duplicates

I would like to come up with a standard practice to prevent any tables from having duplicates where it matters. In most cases duplicates are a combination of variables rather than one. My primary keys are just the unique ids for each field so I cannot use them. What I have been doing is querying the table first and then if the number of rows for the combination in question is 0, making the insert. However, I have read it should be possible to set up a unique key over multiple fields to enforce uniqueness. INSERT IGNORE sounds like a good possibility, however, I would need it to ignore on more

In Python numpy.unique can remove all duplicates from a 1D array, very efficiently. 1) How about to remove duplicate rows or columns in a 2D array ? 2) How about for nD arrays ? If possible I would use pandas. In [1]: from pandas import * In [2]: import numpy as np In [3]: a = np.array([[1, 1], [2, 3], [1, 1], [5, 4], [2, 3]]) In [4]: DataFrame(a).drop_duplicates().values Out[4]: array([[1, 1], [2, 3], [5, 4]], dtype=int64) The following is another approach which performs much better than for loop. 2s for 10k+100 duplicates. def tuples(A): try: return tuple(tuples(a) for a in A) except

I need to make a insert but only if similar record don't exists for example: INSERT INTO requests ('user_id','subject','text','time') VALUES (56,'test','test 1234',6516516) but to check if there are same 'subject' and 'text' in another record to: not insert anything update 'time' and 'user_id' I need sql for both cases because I'm no sure at this moment what I'm going to use. Thanks in advance! INSERT INTO requests ('user_id','subject','text','time') VALUES (56,'test','test 1234',6516516) ON DUPLICATE KEY UPDATE time = VALUES(time), user_id = VALUES(user_id) Have the relevant columns set to

I want to detect duplicate values in a Java array. For example: int[] array = { 3, 3, 3, 1, 5, 8, 11, 4, 5 }; How could I get the specific duplicated entry and how many times it occurs? I'll have a Map<Integer, Integer> where the first integer is the value of the number that occurs in the array and the second integer is the count (number of occurrence). Run through the array.length in a loop for each item in the array, do a map.containsKey(array[i]) . If there exists a number in a map, increment that number (something like map.put(array[i], map.get(array[i]) + 1) . Otherwise, create a new

How do I remove consecutive duplicates from a list like this in python? lst = [1,2,2,4,4,4,4,1,3,3,3,5,5,5,5,5] Having a unique list or set wouldn't solve the problem as there are some repeated values like 1,...,1 in the previous list. I want the result to be like this: newlst = [1,2,4,1,3,5] Would you also please consider the case when I have a list like this [4, 4, 4, 4, 2, 2, 3, 3, 3, 3, 3, 3] and I want the result to be [4,2,3,3] rather than [4,2,3] . itertools.groupby() is your solution. newlst = [k for k, g in itertools.groupby(lst)] If you wish to group and limit the group size by the

This was a question in one my friend's programming class. Q. How do you sort an array of int s and then arrange them such that all duplicate elements appear at the end of the array? For example, given the input {5, 2, 7, 6, 1, 1, 5, 6, 2} The output would be {1, 2, 5, 6, 7, 1, 2, 5, 6} Note that the numbers are sorted and duplicate numbers are after 7, which is the maximum in the array. This has to be achieved with out using any Java library packages/utils . I suggested to sort the array first using insertion or bubble sort, and then go over the array, perform something like the following :

This question already has an answer here: pair-wise duplicate removal from dataframe [duplicate] 4 answers I'm sure it's something simple, but I have a data frame df <- data.frame(a = c(1, 2, 3), b = c(2, 3, 1), c = c(3, 1, 4)) And I want a new data frame that contains the unique combinations of values in the rows, irrespective of which column they're in. So in the case above I'd want a b c 1 2 3 3 1 4 I've tried unique(df[c('a', 'b', 'c')]) but it sees (1, 2, 3) as unique from (2, 3, 1), which I don't want. Maybe something like that indx <- !duplicated(t(apply(df, 1, sort))) # finds non -

I want to remove duplicate rows from a matrix. I read How can I remove duplicates in an array but keep the same order? , but this is not exactly what I want. The solution above removes duplicate values (cells) from matrix (and returns a vector), but I need to remove duplicate rows and return a matrix — the same matrix without duplicate rows. Example: a = [1,2; 3,4; 5,6; 1,2; 7,8] a = 1 2 3 4 5 6 1 2 7 8 %... ans = 1 2 3 4 5 6 7 8 The order doesn't matter. See http://www.mathworks.com/help/techdoc/ref/unique.html b = unique(A, 'rows') returns the unique rows of A. Here is my solution package

I'd like to remove all items that appear more than once in a vector. Specifically, this includes character, numeric and integer vectors. Currently, I'm using duplicated() both forwards and backward (using the fromLast parameter). Is there a more computationally efficient (faster) way to execute this in R? The solution below is simple enough to write/read, but it seems inefficient to execute the duplicate search twice. Perhaps a counting-based method using an additional data structure would be better? Example: d <- c(1,2,3,4,1,5,6,4,2,1) d[!(duplicated(d) | duplicated(d, fromLast=TRUE))] #[1] 3