Cumulative operations on dtype objects

Question

I am trying to figure out how I can apply cumulative functions to objects. For numbers there are several alternatives like cumsum and cumcount. The

Answer 1

Turns out this cannot be done.

Continuing on the same sample:

def burndowntheworld(ser):
    print('Are you sure?')
    return ser/0

df.select_dtypes(['object']).expanding().apply(burndowntheworld)
Out: 
    C2 C3   C4
0  {A}  A  [A]
1  {B}  B  [B]
2  {C}  C  [C]
3  {D}  D  [D]

If the column's type is object, the function is never called. And pandas doesn't have an alternative that works on objects. It's the same for rolling().apply().

In some sense, this is a good thing because expanding.apply with a custom function has O(n**2) complexity. With special cases like cumsum, ewma etc, the recursive nature of the operations can decrease the complexity to linear time but in the most general case it should calculate the function for the first n elements, and then for the first n+1 elements and so on. Therefore, especially for a function which is only dependent on the current value and function's previous value, expanding is quite inefficient. Not to mention storing lists or sets in a DataFrame is never a good idea to begin with.

So the answer is: if your data is not numeric and the function is dependent on the previous result and the current element, just use a for loop. It will be more efficient anyway.