merge two dataframes without repeats pandas

Question

I am trying to merge two dataframes, one with columns: customerId, full name, and emails and the other dataframe with columns: customerId, amount, and date. I want to have the first dataframe be the main dataframe and the other dataframe information be included but only if the customerIds match up; I tried doing:

 merge = pd.merge(df, df2, on='customerId', how='left')

but the dataframe that is produced contains a lot of repeats and looks wrong:

customerId  full name   emails  amount  date
0   002963338   Star shine  star.shine@cdw.com  $2,910.94   2016-06-14
1   002963338   Star shine  star.shine@cdw.com  $9,067.70   2016-05-27
2   002963338   Star shine  star.shine@cdw.com  $6,507.24   2016-04-12
3   002963338   Star shine  star.shine@cdw.com  $1,457.99   2016-02-24
4   986423367   palm tree   tree.palm@snapchat.com,tree@.com    $4,604.83   2016-07-16

this cant be right, please help!

Answer 1

There is problem you have duplicates in customerId column.

So solution is remove them, e.g. by drop_duplicates:

df2 = df2.drop_duplicates('customerId')

Sample:

df = pd.DataFrame({'customerId':[1,2,1,1,2], 'full name':list('abcde')})
print (df)
   customerId full name
0           1         a
1           2         b
2           1         c
3           1         d
4           2         e

df2 = pd.DataFrame({'customerId':[1,2,1,2,1,1], 'full name':list('ABCDEF')})
print (df2)
   customerId full name
0           1         A
1           2         B
2           1         C
3           2         D
4           1         E
5           1         F

---

merge = pd.merge(df, df2, on='customerId', how='left')
print (merge)
    customerId full name_x full name_y
0            1           a           A
1            1           a           C
2            1           a           E
3            1           a           F
4            2           b           B
5            2           b           D
6            1           c           A
7            1           c           C
8            1           c           E
9            1           c           F
10           1           d           A
11           1           d           C
12           1           d           E
13           1           d           F
14           2           e           B
15           2           e           D

df2 = df2.drop_duplicates('customerId')
merge = pd.merge(df, df2, on='customerId', how='left')
print (merge)
   customerId full name_x full name_y
0           1           a           A
1           2           b           B
2           1           c           A
3           1           d           A
4           2           e           B

Answer 2

I do not see repeats as a whole row but there are repetetions in customerId. You could remove them using:

    df.drop_duplicates('customerId', inplace = 1) 

where df could be the dataframe corresponding to amount or one obtained post merge. In case you want fewer rows (say n), you could use:

    df.groupby('customerId).head(n)