Predicting per pixel sparse motion vector

Question

I am researching on sparse pixel level matching between two frames of video. What could be possible approach to predict per pixel motion vector between consecutive frames if