grequests

问题 I'm working with a process which is basically as follows: Take some list of urls. Get a Response object from each. Create a BeautifulSoup object from the text of each Response. Pull the text of a certain tag from that BeautifulSoup object. From my understanding, this seems ideal for grequests: GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. But yet, the two processes (one with requests, one with grequests) seem to be getting me different results,

问题 I need to make 100k head requests, and I'm using gevent on top of requests. My code runs for a while, but then eventually hangs. I'm not sure why it's hanging, or whether it's hanging inside requests or gevent. I'm using the timeout argument inside both requests and gevent. Please take a look at my code snippet below, and let me know what I should change. import gevent from gevent import monkey, pool monkey.patch_all() import requests def get_head(url, timeout=3): try: return requests.head

问题 I'm running an AWS Lambda script with a Python 2.7 runtime. However, whenever it initializes it begins to import the grequests library but fails because of it's dependency on gevent: Gevent is required for grequests. It seems it is successfully finding the grequests library (since it knows it needs gevent) but fails. What I've tried so far: pip install --ignore-installed grequests -t . pip install --ignore-installed grequests -t ./lib pip install --ignore-installed gevent -t . pip install -

问题 I'm running an AWS Lambda script with a Python 2.7 runtime. However, whenever it initializes it begins to import the grequests library but fails because of it's dependency on gevent: Gevent is required for grequests. It seems it is successfully finding the grequests library (since it knows it needs gevent) but fails. What I've tried so far: pip install --ignore-installed grequests -t . pip install --ignore-installed grequests -t ./lib pip install --ignore-installed gevent -t . pip install -

问题 I am very new to all of this; I need to obtain data on several thousand sourceforge projects for a paper I am writing. The data is all freely available in json format at the url http://sourceforge.net/api/project/name/[project name]/json. I have a list of several thousand of these URL's and I am using the following code. import grequests rs = (grequests.get(u) for u in ulist) answers = grequests.map(rs) Using this code I am able to obtain the data for any 200 or so projects I like, i.e. rs =

问题 I am using Python requests in celery workers to make large number of (~10/sec) API calls(includes GET,POST, PUT, DELETE). Each request takes around 5-10s to complete. I tried running celery workers in eventlet pool, with 1000 concurrency. Since requests are blocking process each concurrent connection is waiting on one request. How do I make requests asynchronous? 回答1: Use eventlet monkey patching to make any pure python library non-blocking. patch single library # import requests # instead do

问题 I am using the grequests to asynchronously download data from a website using the same url but different parameters. For example, unsent_requests = [] for param in params: # assume params is a list containing different parameters or query strings unsent_requests.append(grequests.get(url = url, params = param)) responses = grequests.map(unsent) How can I possibly get to know which response from responses belongs to which request from unsent_requests? Or are the responses in the same order as

问题 I have small code snippet as below: import requests import multiprocessing header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'https://my_server.com/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json

问题 I'm hosting a server on localhost and I want to fire hundreds of GET requests asynchronously. For this I am using grequests . Everything appears to work fine but I repeatedly get the warning: WARNING:requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection: date.jsontest.com A search shows how the full pool issue can be avoided when creating a Session() in requests e.g. here. However, a couple of things: Even if I don't take any steps to avoid the warning, I

问题 How can I send like 1000 requests the fastest way? I know that you can send multiple request with grequests : urls = [ 'sample.url/1', 'sample.url/2', ... ] request = (grequests.get(u) for u in urls) print grequests.map(request) But the return is not the content. What I need is to get the json data, so for example something like this: request = (grequests.get(u) for u in urls) content = grequests.json(request) 回答1: The items returned are not the content, but they do include the content. You