data.table cumulative stats of irregular observations with time window

Question

I have some transactional records, like the following:

library(data.table)
customers      <- 1:75
purchase_dates <- seq( as.Date('2016-01-01'),
                       as.Date('2018-12-31'), 
                       by=1 )
n <- 500L

set.seed(1)

# Assume the data are already ordered and 1 row per cust_id/purch_dt
df <- data.table( cust_id   = sample(customers, n, replace=TRUE),
                  purch_dt  = sample(purchase_dates, n, replace=TRUE),
                  purch_amt = sample(500:50000, n, replace=TRUE)/100
                  )[, .(purch_amt = sum(purch_amt)), 
                      keyby=.(cust_id, purch_dt) ]
df
# cust_id   purch_dt purch_amt
#       1 2016-03-20     69.65
#       1 2016-05-17    413.60
#       1 2016-12-25    357.18
#       1 2017-03-20    256.21
#       2 2016-05-26     49.14
#       2 2018-05-31    261.87
#       2 2018-12-27    293.28
#       3 2016-12-10    204.12
#       3 2018-09-21      8.70

I would like to know the prior transaction count and total amount, within a 365-day prior window (i.e., at d-365 through d-1 for a transaction on date d).

I thought of using a rolling join, but that would match to at most one prior purchase, and there could be multiple purchases.

I was able to get the desired result using a Cartesian self-join with a date filter (see answer below), but that's not a very memory-efficient approach.

Desired output:

 cust_id   purch_dt prior_purch_cnt prior_purch_amt purch_amt
       1 2016-03-20               0            0.00     69.65
       1 2016-05-17               1           69.65    413.60
       1 2016-12-25               2          483.25    357.18
       1 2017-03-20               3          840.43    256.21
       2 2016-05-26               0            0.00     49.14
       2 2018-05-31               0            0.00    261.87
       2 2018-12-27               1          261.87    293.28
       3 2016-12-10               0            0.00    204.12
       3 2018-09-21               0            0.00      8.70

Answer 1

I would like to know the prior transaction count and total amount, within a 365-day prior window (i.e., at d-365 through d-1 for a transaction on date d).

I think the idiomatic way is:

df[, c("ppn", "ppa") := 
  df[.(cust_id = cust_id, d_dn = purch_dt-365, d_up = purch_dt), 
    on=.(cust_id, purch_dt >= d_dn, purch_dt < d_up), 
    .(.N, sum(purch_amt, na.rm=TRUE))
  , by=.EACHI][, .(N, V2)]
]

     cust_id   purch_dt purch_amt ppn    ppa
  1:       1 2016-03-20     69.65   0   0.00
  2:       1 2016-05-17    413.60   1  69.65
  3:       1 2016-12-25    357.18   2 483.25
  4:       1 2017-03-20    256.21   3 840.43
  5:       2 2016-05-26     49.14   0   0.00
 ---                                        
494:      75 2018-01-12    381.24   2 201.04
495:      75 2018-04-01     65.83   3 582.28
496:      75 2018-06-17    170.30   4 648.11
497:      75 2018-07-22     60.49   5 818.41
498:      75 2018-10-10     66.12   4 677.86

This is a "non-equi join".

Answer 2

Here's the Cartesian self-join with date-range filter:

df_prior <- df[df, on=.(cust_id), allow.cartesian=TRUE
                ][i.purch_dt < purch_dt & 
                    i.purch_dt >= purch_dt - 365
                  ][, .(prior_purch_cnt = .N, 
                        prior_purch_amt = sum(i.purch_amt)),
                     keyby=.(cust_id, purch_dt)]

df2 <- df_prior[df, on=.(cust_id, purch_dt)]

df2[is.na(prior_purch_cnt), `:=`(prior_purch_cnt=0,
                                 prior_purch_amt=0
                                 )]
df2
# cust_id   purch_dt prior_purch_cnt prior_purch_amt purch_amt
#       1 2016-03-20               0            0.00     69.65
#       1 2016-05-17               1           69.65    413.60
#       1 2016-12-25               2          483.25    357.18
#       1 2017-03-20               3          840.43    256.21
#       2 2016-05-26               0            0.00     49.14

I'm concerned about how this could blow up prior to filtering on datasets where customers have many prior transactions.