When using scrapy-splash library to render JS. We add its custom DUPEFILTER_CLASS to the settings.py file.
DUPEFILTER_CLASS = \'scrapy_splash.SplashAwareDupeF
