pandas DataFrame concat / update (“upsert”)?

Question

I am looking for an elegant way to append all the rows from one DataFrame to another DataFrame (both DataFrames having the same index and column structure), but in cases whe

Answer 1

One solution is to conatenate df1 with new rows in df2 (i.e. where the index does not match). Then update the values with those from df2.

df = pd.concat([df1, df2[~df2.index.isin(df1.index)]])
df.update(df2)

>>> df
             A   B
2015-10-01  A1  B1
2015-10-02  a1  b1
2015-10-03  a2  b2
2015-10-04  a3  b3

EDIT: Per the suggestion of @chrisb, this can further be simplified as follows:

pd.concat([df1[~df1.index.isin(df2.index)], df2])

Thanks Chris!

Answer 2

In addition to the correct answer, watch out for columns that do not exist in both dataframes:

    df1 = pd.DataFrame([['test',1, True], ['test2',2, True]]).set_index(0)
    df2 = pd.DataFrame([['test2',4], ['test3',3]]).set_index(0)

If you just use the aforementioned solution as-is, you get:

    >>>     1   2
    0       
    test    1   True
    test2   4   NaN
    test3   3   NaN

But if you are expecting the following output:

    >>>     1   2
    0       
    test    1   True
    test2   4   True
    test3   3   NaN

Just change the statement to:

    df1 = pd.concat([df1, df2[~df2.index.isin(df1.index)]])
    df1.update(df2)