This is because you did not set up correctly to run Firefox without GUI.
This is a tutorial that might be helpful
Selenium Headless Automated Testing in Ubuntu
I think the reason is you need to specify the port number, probably xvfb runs on a different port than Firefox
on one terminal:
xvfb :99 -ac
on the another terminal:
export DISPLAY=:99
and run your scrapy program
I am successful without using root
