Efficient filtering of a large dataset in Spark based on other dataset

Question

I have an issue with filtering a large dataset, which I believe is because of an inefficient join. What I\'m trying to do is the following:

Dataset info cont