urllib2

python urllib2 basic authentication

懵懂的女人 提交于 2019-12-01 05:59:46
Hi i'm trying to use python to access an API URL using urllib2: import urllib2 url = 'https://XXXXXXXXXX.com/' username = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' password = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, url, username, password) authhandler = urllib2.HTTPBasicAuthHandler(passman) opener = urllib2.build_opener(authhandler) urllib2.install_opener(opener) pagehandle = urllib2.urlopen(url) I don't know what the realm is but am assuming I can use default i.e. None. Anyway, i am still get 401 error: Traceback

Download a file from https with authentication

六月ゝ 毕业季﹏ 提交于 2019-12-01 05:53:47
I have a Python 2.6 script that downloades a file from a web server. I want this this script to pass a username and password(for authenrication before fetching the file) and I am passing them as part of the url as follows: import urllib2 response = urllib2.urlopen("http://'user1':'password'@server_name/file") However, I am getting syntax error in this case. Is this the correct way to go about it? I am pretty new to Python and coding in general. Can anybody help me out? Thanks! I suppose you are trying to pass through a Basic Authentication. In this case, you can handle it this way: import

Why urllib returns garbage from some wikipedia articles?

十年热恋 提交于 2019-12-01 05:31:21
问题 >>> import urllib2 >>> good_article = 'http://en.wikipedia.org/wiki/Wikipedia' >>> bad_article = 'http://en.wikipedia.org/wiki/India' >>> req1 = urllib2.Request(good_article) >>> req2 = urllib2.Request(bad_article) >>> req1.add_header('User-Agent', 'Mozilla/5.0') >>> req2.add_header('User-Agent', 'Mozilla/5.0') >>> result1 = urllib2.urlopen(req1) >>> result2 = urllib2.urlopen(req2) >>> result1.readline() '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR

Close urllib2 connection

[亡魂溺海] 提交于 2019-12-01 03:36:49
I'm using urllib2 to load files from ftp- and http-servers. Some of the servers support only one connection per IP. The problem is, that urllib2 does not close the connection instantly. Look at the example-program. from urllib2 import urlopen from time import sleep url = 'ftp://user:pass@host/big_file.ext' def load_file(url): f = urlopen(url) loaded = 0 while True: data = f.read(1024) if data == '': break loaded += len(data) f.close() #sleep(1) print('loaded {0}'.format(loaded)) load_file(url) load_file(url) The code loads two files (here the two files are the same) from an ftp-server which

python urllib使用

随声附和 提交于 2019-12-01 01:58:13
简介: urllib2是python的一个获取url(Uniform Resource Locators,统一资源定址器)的模块。它用urlopen函数的形式提供了一个非常简洁的接口。这使得用各种各样的协议获取url成为可能。它同时 也提供了一个稍微复杂的接口来处理常见的状况-如基本的认证,cookies,代理,等等。这些都是由叫做opener和handler的对象来处理的。 以下是获取url最简单的方式: import urllib2 response = urllib2.urlopen('http://python.org/') html = response.read() 许多urlib2的使用都是如此简单(注意我们本来也可以用一个以”ftp:”"file:”等开头的url取代”HTTP”开头的url).然 而,这篇教程的目的是解释关于HTTP更复杂的情形。HTTP建基于请求和回应(requests &responses )-客户端制造请求服务器返回回应。urlib2用代 表了你正在请求的HTTP request的Request对象反映了这些。用它最简单的形式,你建立了一个Request对象来明确指明你想要获取的url。调用urlopen函 数对请求的url返回一个respons对象。这个respons是一个像file的对象,这意味着你能用.read(

python模块之---- urllib2模块详解

爱⌒轻易说出口 提交于 2019-12-01 01:57:54
简介: urllib2是python的一个获取url(Uniform Resource Locators,统一资源定址器)的模块。它用urlopen函数的形式提供了一个非常简洁的接口。这使得用各种各样的协议获取url成为可能。它同时 也提供了一个稍微复杂的接口来处理常见的状况-如基本的认证,cookies,代理,等等。这些都是由叫做opener和handler的对象来处理的。 以下是获取url最简单的方式: import urllib2 response = urllib2.urlopen('http://python.org/') html = response.read() 许多urlib2的使用都是如此简单(注意我们本来也可以用一个以”ftp:”"file:”等开头的url取代”HTTP”开头的url).然 而,这篇教程的目的是解释关于HTTP更复杂的情形。HTTP建基于请求和回应(requests &responses )-客户端制造请求服务器返回回应。urlib2用代 表了你正在请求的HTTP request的Request对象反映了这些。用它最简单的形式,你建立了一个Request对象来明确指明你想要获取的url。调用urlopen函 数对请求的url返回一个respons对象。这个respons是一个像file的对象,这意味着你能用.read(

How to download a webpage that require username and password?

大兔子大兔子 提交于 2019-12-01 01:40:14
For example, I want to download this page after inserting username and password: http://forum.ubuntu-it.org/ I have tryed with wget but doesn't work. Is there a solution with python ? You can test with these username and password: username: johnconnor password: hellohello Like @robert says, use mechanize. To get you started: from mechanize import Browser b = Browser() b.open("http://forum.ubuntu-it.org/index.php") b.select_form(nr=0) b["user"] = "johnconnor" b["passwrd"] = "hellohello" b.submit() response = b.response().read() if "Salve <b>johnconnor</b>" in response: print "Logged in!" Try

List all files in an online directory with Python?

∥☆過路亽.° 提交于 2019-12-01 01:07:42
Hello i was just wondering i'm trying to create a python application that downloads files from the internet but at the moment it only downloads one file with the name i know... is there any way that i can get a list of files in an online directory and downloaded them? ill show you my code for downloading one file at a time, just so you know a bit about what i wan't to do. import urllib2 url = "http://cdn.primarygames.com/taxi.swf" file_name = url.split('/')[-1] u = urllib2.urlopen(url) f = open(file_name, 'wb') meta = u.info() file_size = int(meta.getheaders("Content-Length")[0]) print

Python urllib2 resume download doesn't work when network reconnects

China☆狼群 提交于 2019-12-01 00:52:07
I'm using urllib2 to make a resuming downloader, roughly based on this method. I can end the program and re-start it, and it starts downloading where it left off, downloading the file that ends up the same size as if it were downloaded all at once. However, I have tested it when disabling and reenabling network, and it doesn't download correctly. The file size ends up longer than the file should be, and the file doesn't work correctly. Is there something I missed, or could this be a urllib2 bug? import urllib2 opener = urllib2.build_opener(); self.count = 0 # Counts downloaded size. self

Trying to post multipart form data in python, won't post

雨燕双飞 提交于 2019-11-30 22:05:28
I'm fairly new to python, so I apologize in advance if this is something simple I'm missing. I'm trying to post data to a multipart form in python. The script runs, but it won't post. I'm not sure what I'm doing wrong. import urllib, urllib2 from poster.encode import multipart_encode from poster.streaminghttp import register_openers def toqueXF(): register_openers() url = "http://localhost/trunk/admin/new.php" values = {'form':open('/test.pdf'), 'bandingxml':open('/banding.xml'), 'desc':'description'} data, headers = multipart_encode(values) request = urllib2.Request(url, data, headers)