urllib库:分析Robots协议

 1from urllib.robotparser import RobotFileParser 2import ssl 3from urllib.request import urlopen 4ssl._create_default_https_context = ssl._create_unverified_context 5 6rp = RobotFileParser() 7rp.set_url('http://www.jianshu.com/robots.txt') 8rp.read() 9print(rp.can_fetch('*', 'http://www.jianshu.com/p/b6755402d7d'))10print(rp.can_fetch('*', 'http://www.jianshu.com/search?q=python&page=1&type=note'))

parse()读取分析

1rp = RobotFileParser()2rp.parse(urlopen('http://www.jianshu.com/robots.txt').read().decode('utf-8').split('\n'))

`

来源：https://www.cnblogs.com/gxj521test/p/10206559.html