beautifulsoup

Python爬虫实战，60行代码爬取英雄联盟全英雄全皮肤，找寻曾今那些被删除的绝版皮肤

由 橙三吉。 提交于 2019-12-28 16:00:58
【推荐】2019 Java 开发者跳槽指南.pdf(吐血整理) >>> 学了一周多的爬虫课后终于按捺不住了，小编决定自己手动编写爬虫程序，刚好LJ在鼓励学员分享成果，优秀作品有奖励，就把自己用Python编程爬取各大游戏高清壁纸的过程整理了出来进行投稿，与大家一起分享。 爬取了当前比较火的游戏壁纸，MOBA游戏《英雄联盟》，手游《王者荣耀》、《阴阳师》，FPS游戏《绝地求生》，其中《英雄联盟》的壁纸最难爬取，这里展示爬取《英雄联盟》全部英雄壁纸的过程，学会了这个，自己再去爬取其他游戏壁纸也就不成问题啦。 先看一下最终爬取的效果，每个英雄的壁纸都被爬取下来了： “黑暗之女 安妮”的12张壁纸： 高清大图： 下面开始正式教学！ 版本：Python 3.5 工具：Jupyter notebook实现各个环节，最终整合成LOL_scrawl.py文件 1.了解爬取对象，设计爬取流程 在使用爬虫前，先花一定时间对爬取对象进行了解，是非常有必要的，这样可以帮助我们科学合理地设计爬取流程，以避开爬取难点，节约时间。 1.1英雄基本信息 打开英雄联盟官网，看到所有英雄的信息： 若要爬取全部英雄，我们先要获取这些英雄的信息，在网页上“右击——检查——Elements”，就能在看到英雄的信息了，如下图所示，包括英雄昵称、英雄名称、英文名等等。由于这些信息是使用JavaScript动态加载的

Wait page to load before getting data with requests.get in python 3

由 核能气质少年 提交于 2019-12-28 12:28:29
问题 I have a page that i need to get the source to use with BS4, but the middle of the page takes 1 second(maybe less) to load the content, and requests.get catches the source of the page before the section loads, how can I wait a second before getting the data? r = requests.get(URL + self.search, headers=USER_AGENT, timeout=5 ) soup = BeautifulSoup(r.content, 'html.parser') a = soup.find_all('section', 'wrapper') The page <section class="wrapper" id="resultado_busca"> 回答1: It doesn't look like a

Web scraping a website with dynamic javascript content

由 可紊 提交于 2019-12-28 05:56:25
问题 So I'm using python and beautifulsoup4(which i'm not tied to) to scrape a website. Problem is when I use urlib to grab the html of a page it's not the entire page because some of it is generated via the javascript. Is there any way to get around this? 回答1: There are basically two main options to proceed with: using browser developer tools, see what ajax requests are going to load the page and simulate them in your script, you will probably need to use json module to load the response json

How to change tag name with BeautifulSoup?

由 柔情痞子 提交于 2019-12-28 04:18:27
问题 I am using python + BeautifulSoup to parse an HTML document. Now I need to replace all <h2 class="someclass"> elements in an HTML document, with <h1 class="someclass"> . How can I change the tag name, without changing anything else in the document? 回答1: I don't know how you're accessing tag but the following works for me: import BeautifulSoup if __name__ == "__main__": data = """ <html> <h2 class='someclass'>some title</h2> <ul> <li>Lorem ipsum dolor sit amet, consectetuer adipiscing elit.<

How to find tags with only certain attributes - BeautifulSoup

由 本秂侑毒 提交于 2019-12-28 03:31:40
问题 How would I, using BeautifulSoup, search for tags containing ONLY the attributes I search for? For example, I want to find all <td valign="top"> tags. The following code: raw_card_data = soup.fetch('td', {'valign':re.compile('top')}) gets all of the data I want, but also grabs any <td> tag that has the attribute valign:top I also tried: raw_card_data = soup.findAll(re.compile('<td valign="top">')) and this returns nothing (probably because of bad regex) I was wondering if there was a way in

How to extract and download all images from a website using beautifulSoup?

由 故事扮演 提交于 2019-12-28 02:04:39
问题 I am trying to extract and download all images from a url. I wrote a script import urllib2 import re from os.path import basename from urlparse import urlsplit url = "http://filmygyan.in/katrina-kaifs-top-10-cutest-pics-gallery/" urlContent = urllib2.urlopen(url).read() # HTML image tag: <img src="url" alt="some_text"/> imgUrls = re.findall('img .*?src="(.*?)"', urlContent) # download all images for imgUrl in imgUrls: try: imgData = urllib2.urlopen(imgUrl).read() fileName = basename(urlsplit

How to extract and download all images from a website using beautifulSoup?

由 ℡╲_俬逩灬. 提交于 2019-12-28 02:04:26
问题 I am trying to extract and download all images from a url. I wrote a script import urllib2 import re from os.path import basename from urlparse import urlsplit url = "http://filmygyan.in/katrina-kaifs-top-10-cutest-pics-gallery/" urlContent = urllib2.urlopen(url).read() # HTML image tag: <img src="url" alt="some_text"/> imgUrls = re.findall('img .*?src="(.*?)"', urlContent) # download all images for imgUrl in imgUrls: try: imgData = urllib2.urlopen(imgUrl).read() fileName = basename(urlsplit

Python: BeautifulSoup - get an attribute value based on the name attribute

由 主宰稳场 提交于 2019-12-28 01:42:28
问题 I want to print an attribute value based on its name, take for example <META NAME="City" content="Austin"> I want to do something like this soup = BeautifulSoup(f) //f is some HTML containing the above meta tag for meta_tag in soup('meta'): if meta_tag['name'] == 'City': print meta_tag['content'] The above code give a KeyError: 'name' , I believe this is because name is used by BeatifulSoup so it can't be used as a keyword argument. 回答1: It's pretty simple, use the following - >>> from bs4

How to load all entries in an infinite scroll at once to parse the HTML in python

由 一曲冷凌霜 提交于 2019-12-27 19:14:11
问题 I am trying to extract information from this page. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the HTML and get the information that I need for the first 10 entries, but I want to fully load all entries before parsing the HTML. I am using python, requests, and BeautifulSoup. The way I parse the page when it loads with the first 10 entries is as follows: from bs4 import BeautifulSoup import requests s = requests.Session()

How to load all entries in an infinite scroll at once to parse the HTML in python

由 爱⌒轻易说出口 提交于 2019-12-27 19:13:58
问题 I am trying to extract information from this page. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the HTML and get the information that I need for the first 10 entries, but I want to fully load all entries before parsing the HTML. I am using python, requests, and BeautifulSoup. The way I parse the page when it loads with the first 10 entries is as follows: from bs4 import BeautifulSoup import requests s = requests.Session()
订阅 beautifulsoup