Spatial and temporal averaging on depth image

Question

I would like to do Spatial and temporal on a depth image. The spatial average can be achieved by convolution (cv::filter2D), for temporal average you would need