Python-Requests, extract url parameters from a string

Question

I am using this awesome library called requests to maintain python 2 & 3 compatibility and simplify my application requests management.

I have a case where I nee

Answer 1

Using requests only:

query = requests.utils.urlparse(url).query
params = dict(x.split('=') for x in query.split('&'))

if 'token' in params:
    params['token'] = 'NEW_TOKEN'

requests.get(url, params=params)

Answer 2

You cannot use requests for this; the library builds such URLs if passed a Python structure for the parameters, but does not offer any tools to parse them. That's not a goal of the project.

Stick to the urllib.parse method to parse out the parameters. Once you have a dictionary or list of key-value tuples, just pass that to requests to build the URL again:

try:
    # Python 3
    from urllib.parse import urlparse, parse_qs
except ImportError:
    # Python 2
    from urlparse import urlparse, parse_qs

o = urlparse(url)
query = parse_qs(o.query)
# extract the URL without query parameters
url = o._replace(query=None).geturl()

if 'token' in query:
    query['token'] = 'NEW_TOKEN'

requests.get(url, params=query)

You can get both the urlparse and parse_qs functions in both Python 2 and 3, all you need to do is adjust the import location if you get an exception.

Demo on Python 3 (without the import exception guard) to demonstrate the URL having been built:

>>> from urllib.parse import urlparse, parse_qs
>>> url = "http://httpbin.org/get?token=TOKEN_TO_REPLACE&param2=c"
>>> o = urlparse(url)
>>> query = parse_qs(o.query)
>>> url = o._replace(query=None).geturl()
>>> if 'token' in query:
...     query['token'] = 'NEW_TOKEN'
... 
>>> response = requests.get(url, params=query)
>>> print(response.text)
{
  "args": {
    "param2": "c", 
    "token": "NEW_TOKEN"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/3.4.2 Darwin/14.1.0"
  }, 
  "origin": "188.29.165.245", 
  "url": "http://httpbin.org/get?token=NEW_TOKEN&param2=c"
}