Putting a `Cookie` in a `CookieJar`

Question

I\'m using the new Python Requests library to make http requests. I obtain a cookie from the server as text. How do I turn that into a CookieJar with the cookie

Answer 1

To help you out, I wrote an entire module. I tried it with my personal webpage and google's cookies, so I'd assume it works.

I got help from How to add cookie to existing cookielib CookieJar instance in Python?

I have a lot of unpythonic code in here including a semi-kludge, so your mileage may vary. Tweak it as you wish, especially with the assumed items (such as port 80) the "request" as an argument below is of type requests.request and I realized that the "method" argument must be all capitals. Hope I could help!

Note: I haven't had time to add comments for clarification so you'll have to use the source.

import Cookie,cookielib,requests,datetime,time  #had this out but realized later I needed it when I continued testing

def time_to_tuple(time_string):
    wday = {'Mon':0,'Tue':1,'Wed':2,'Thu':3,'Fri':4,'Sat':5,'Sun':6}
    mon = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':11,'Dec':12}
    info = time_string.split(' ')
    info = [i.strip() for i in info if type(i)==str]
    month = None
    for i in info:
        if '-' in i:
            tmp = i.split('-')
            for m in tmp:
                try:
                    tmp2 = int(m)
                    if tmp2<31:
                        mday = tmp2
                    elif tmp2 > 2000:
                        year = tmp2
                except:
                    for key in mon:
                        if m.lower() in key.lower():
                            month = mon[key]
        elif ':' in i:
            tmp = i.split(':')
            if len(tmp)==2:
                hour = int(tmp[0])
                minute = int(tmp[1])
            if len(tmp)==3:
                hour = int(tmp[0])
                minute = int(tmp[1])
                second = int(tmp[2])
        else:
            for item in wday:
                if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
                    day = wday[item]
            if month is None:
                for item in mon:
                    if ((i.lower() in item.lower()) or (item.lower() in i.lower())):
                        month = mon[item]
    return year,month,mday,hour,minute,second

def timefrom(year,month,mday,hour,minute,second):
    time_now = time.gmtime()
    datetime_now = datetime.datetime(time_now.tm_year,time_now.tm_mon,
                                     time_now.tm_mday,time_now.tm_hour,
                                     time_now.tm_min,time_now.tm_sec)
    then = datetime.datetime(year,month,mday,hour,minute,second)
    return (datetime_now-then).total_seconds()

def timeto(year,month,mday,hour,minute,second):
    return -1*timefrom(year,month,mday,hour,minute,second)



##['comment', 'domain', 'secure', 'expires', 'max-age', 'version', 'path', 'httponly']
def parse_request(request):
    headers = request.headers
    cookieinfo = headers['set-cookie'].split(';')
    name = 'Undefined'
    port=80
    port_specified=True
    c = Cookie.SmartCookie(headers['set-cookie'])
    cj = cookielib.CookieJar()
    for m in c.values():
        value = m.coded_value
        domain = m['domain']
        expires = m['expires']
        if type(expires) == str:
            tmp = time_to_tuple(expires)
            expires = timeto(tmp[0],tmp[1],tmp[2],tmp[3],tmp[4],tmp[5])
        max_age=m['max-age']
        version = m['version']
        if version == '':
            version = 0
        path = m['path']
        httponly = m['httponly']
        if httponly == '':
            if 'httponly' in headers['set-cookie'].lower():
                httponly = True
        else:
            httponly = False
        secure = m['secure']
        comment=m['comment']
        port = 80
        port_specified=False
        domain_specified=True
        domain_initial_dot = domain.startswith('.')
        path_specified=True
        discard = True
        comment_url=None
        rest={'HttpOnly':httponly}
        rfc2109=False
        ck = cookielib.Cookie(version,name,value,port,port_specified,domain,
                              domain_specified,domain_initial_dot,path,path_specified,
                              secure,expires,discard,comment,comment_url,rest,rfc2109)
        cj.set_cookie(ck)
    return cj

Answer 2

As dstanek answered, requests will automatically put response cookies in a cookie jar for you.
However, if you manually specify a Cookie header entry, requests will not put those cookies in a jar for you. This means any subsequent requests will be lacking your initial set of cookies, but will have any new cookies going forward.

If you do need to manually create a cookie jar for requests, use requests.cookies.RequestsCookieJar. In case their example code changes:

jar = requests.cookies.RequestsCookieJar()
jar.set('tasty_cookie', 'yum',   domain='httpbin.org', path='/cookies')
jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/elsewhere')
url = 'http://httpbin.org/cookies'
r = requests.get(url, cookies=jar)

Note that if you provide a cookie jar and a Cookie header, the header takes precedence, but the cookie jar will still be maintained for future requests.

Answer 3

Well cookielib.LWPCookieJar has load and save methods on it. Look at the format and see if it matches the native cookie format, you may well be able to load your cookie straight into a cookie jar using StringIO. Alternatively, if requests is using urllib2 under the hood couldn't you add a cookie handler to the default opener?

Answer 4

A requests Session will also receive and send cookies.

s = requests.Session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

print(r.text)
# '{"cookies": {"sessioncookie": "123456789"}}'

(Code above stolen from http://www.python-requests.org/en/latest/user/advanced/#session-objects)

If you want cookies to persist on disk between runs of your code, you can directly use a cookie jar and save/load them. More cumbersome, but still pretty easy:

import requests
import cookielib

cookie_file = '/tmp/cookies'
cj = cookielib.LWPCookieJar(cookie_file)

# Load existing cookies (file might not yet exist)
try:
    cj.load()
except:
    pass

s = requests.Session()
s.cookies = cj

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

# Save cookies to disk, even session cookies
cj.save(ignore_discard=True)

Then look in the file:

$ cat /tmp/cookies 
#LWP-Cookies-2.0
Set-Cookie3: sessioncookie=123456789; path="/"; domain="httpbin.org"; path_spec; discard; version=0