u can use cvblobslib library for blob detection ...
if your inter-frame blob movement is less than the inter blob distance..that is blob displacement is less than the inter blob distance then you can create a list and keep adding the blob in each current frame which fall in the neighbourhood of the blobs in the previous frame...
if your blobs have some constant features like ellipticity...aspect ratio(after fitting a bounding box to it) you can group the blobs with these features into a list..