问题
I have got a very strange problem when I crawl google search engine with wget, curl or python on my servers. Google redirects me to an address starting with [ipv4|ipv6].google.fr/sorry/IndexRedirect... and finally send a 503 error, service unavailable...
Sometimes crawl works correctly and sometimes not during the day, and I tried almost everything possible : forcing ipv4/ipv6 instead of hostname, referer, user agent, vpn, .com/.fr/, proxies and tor, ...
I guess this is an error from Google Servers... any idea ? thanks !
wget "http://google.fr/search?q=test"
--2015-06-03 10:19:52-- http://google.fr/search?q=test
Resolving google.fr (google.fr)... 2a00:1450:400c:c05::5e, 173.194.67.94
Connecting to google.fr (google.fr)|2a00:1450:400c:c05::5e|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://ipv6.google.com/sorry/IndexRedirect?continue=http://google.fr/search%3Fq%3Dtest&q=CGMSECABQdAAUQABAAAAAAAAH1QYqPG6qwUiGQDxp4NLQuHgP_i-oiUu0ZShPumAZRF3u_0 [following]
--2015-06-03 10:19:53-- http://ipv6.google.com/sorry/IndexRedirect?continue=http://google.fr/search%3Fq%3Dtest&q=CGMSECABQdAAUQABAAAAAAAAH1QYqPG6qwUiGQDxp4NLQuHgP_i-oiUu0ZShPumAZRF3u_0
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:400c:c05::64
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:400c:c05::64|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-06-03 10:19:53 ERROR 503: Service Unavailable.
回答1:
Google have triggers to sniff out bots and other abuse of their Terms of Service, so they set a limit (or a "throttle") on the number of calls that the same i.p. address can make over a certain period of time. I believe it's something like 10 calls per minute. Case in point: If you paste your Url into a browser when it fails with a 503 error, you'll get a Captcha challenge from Google to prove you are not a bot.
I am using the pattern.web module to do essentially the same thing as you are doing (for harmless research purposes, of course!), and the documentation for that library shows the throttling limits for most popular APIs (Google, Bing, Twitter, Facebook...).
Try sending your requests every 15+ seconds or so, to avoid tripping up the throttle limit.
来源：https://stackoverflow.com/questions/30614919/google-crawl-503-service-unavailable