TLDR; my question is on how to load compressed video frames from TFRecords.
I am setting up a data pipeline for training deep learning mode
Since you're using very similar dependencies, I suggest to take a look at the following Python package as it addresses your exact problem setting:
pip install video2tfrecord
or refer to https://github.com/ferreirafabio/video2tfrecord.
It should also be adaptable enough to use tf.data.Dataset.
disclaimer: I am one of the authors of the package.