问题
Some servers have a robots.txt file in order to stop web crawlers from crawling through their websites. Is there a way to make a web crawler ignore the robots.txt file? I am using Mechanize for python.
回答1:
The documentation for mechanize has this sample code:
br = mechanize.Browser()
....
# Ignore robots.txt. Do not do this without thought and consideration.
br.set_handle_robots(False)
That does exactly what you want.
回答2:
This looks like what you need:
from mechanize import Browser
br = Browser()
# Ignore robots.txt
br.set_handle_robots( False )
but you know what you're doing…
来源：https://stackoverflow.com/questions/8386481/web-crawler-ignore-robots-txt-file