Is there an inherent recency bias in deep learning?

Question

When working with very large models within Deep Learning, training often takes long and requires small batch sizes due to memory restrictions.
Usually, we are left wit