How to add a third party Scrapy middleware

问题

I'm working with scrapy 1.1 . I want to add scrapy-fake-user-agent Scrapy middleware that would rotate user agents seamlessly and randomly. User Agent strings are supplied by the fake-useragent module.

following the directions from the site, I have:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

However when I run it I get:

ImportError: No module named scrapy_fake_useragent.middleware

What am I doing wrong?

edit: I opened a python shell in the virtualenv:

In[4]: import scrapy.downloadermiddlewares as x
In[5]: x.useragent.UserAgentMiddleware
Out[5]: scrapy.downloadermiddlewares.useragent.UserAgentMiddleware

In[7]: import scrapy_fake_useragent as x
In[8]: dir(x)
Out[8]: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

So 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware 'seems to be available but not 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware' ?

回答1:

I too had the same problem. Later I noticed that I am not inside the virutalenv. Those packages were installed in virutalenv. When moved into virtualenv, it is fine.

来源：https://stackoverflow.com/questions/39430264/how-to-add-a-third-party-scrapy-middleware