lxml

Scrapy crawl with next page

由 独自空忆成欢 提交于 2019-11-30 08:40:07
问题 I have this code for scrapy framework: # -*- coding: utf-8 -*- import scrapy from scrapy.contrib.spiders import Rule from scrapy.linkextractors import LinkExtractor from lxml import html class Scrapy1Spider(scrapy.Spider): name = "scrapy1" allowed_domains = ["sfbay.craigslist.org"] start_urls = ( 'http://sfbay.craigslist.org/search/npo', ) rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse", follow= True),) def parse(self, response): site =

Parse xml with lxml - extract element value

由 霸气de小男生 提交于 2019-11-30 08:22:10
问题 Let's suppose we have the XML file with the structure as follows. <?xml version="1.0" ?> <searchRetrieveResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/zing/srw/ http://www.loc.gov/standards/sru/sru1-1archive/xml-files/srw-types.xsd" xmlns="http://www.loc.gov/zing/srw/"> <records xmlns:ns1="http://www.loc.gov/zing/srw/"> <record> <recordData> <record xmlns=""> <datafield tag="000"> <subfield code="a">123</subfield> <subfield code="b">456<

lxml memory usage when parsing huge xml in python

由 一世执手 提交于 2019-11-30 07:33:30
I am a python newbie. I am trying to parse a huge xml file in my python module using lxml. In spite of clearing the elements at the end of each loop, my memory shoots up and crashes the application. I am sure I am missing something here. Please helpme figure out what that is. Following are main functions I am using - from lxml import etree def parseXml(context,attribList): for _, element in context: fieldMap={} rowList=[] readAttribs(element,fieldMap,attribList) readAllChildren(element,fieldMap,attribList) for row in rowList: yield row element.clear() def readAttribs(element,fieldMap

Preserving original doctype and declaration of an lxml.etree parsed xml

由 痞子三分冷 提交于 2019-11-30 07:25:53
问题 I'm using python's lxml and I'm trying to read an xml document, modify and write it back but the original doctype and xml declaration disappears. I'm wondering if there's an easy way of putting it back in whether through lxml or some other solution? 回答1: tl;dr # adds declaration with version and encoding regardless of # which attributes were present in the original declaration # expects utf-8 encoding (encode/decode calls) # depending on your needs you might want to improve that from lxml

How do I get the whole content between two xml tags in Python?

由 回眸只為那壹抹淺笑 提交于 2019-11-30 06:59:38
问题 I try to get the whole content between an opening xml tag and it's closing counterpart. Getting the content in straight cases like title below is easy, but how can I get the whole content between the tags if mixed-content is used and I want to preserve the inner tags ? <?xml version="1.0" encoding="UTF-8"?> <review> <title>Some testing stuff</title> <text sometimes="attribute">Some text with <extradata>data</extradata> in it. It spans <sometag>multiple lines: <tag>one</tag>, <tag>two</tag> or

Parsing a table with rowspan and colspan

由 梦想的初衷 提交于 2019-11-30 06:31:39
问题 I have a table that I need to parse, specifically it is a school schedule with 4 blocks of time, and 5 blocks of days for every week. I've attempted to parse it, but honestly have not gotten very far because I am stuck with how to deal with rowspan and colspan attributes, because they essentially mean there is a lack of data that I need to continue. As an example of what I want to do, here's a table: <tr> <td colspan="2" rowspan="4">#1</td> <td rowspan="4">#2</td> <td rowspan="2">#3</td> <td

How to find XML Elements via XPath in Python in a namespace-agnostic way?

由 瘦欲@ 提交于 2019-11-30 06:26:20
问题 since I had this annoying issue for the 2nd time, I thought that asking would help. Sometimes I have to get Elements from XML documents, but the ways to do this are awkward. I’d like to know a python library that does what I want, a elegant way to formulate my XPaths, a way to register the namespaces in prefixes automatically or a hidden preference in the builtin XML implementations or in lxml to strip namespaces completely. Clarification follows unless you already know what I want :) Example

爬虫第一天

由 隐身守侯 提交于 2019-11-30 06:23:27
- 张晓波 15027900535 https://www.cnblogs.com/bobo-zhang/ - 爬虫: - requests - 基本使用 - 数据解析 - requets模块的高级 - 单线程+多任务异步协程 - selenium - scrapy框架 - 常用且重要的功能 - 数据分析: - numpy - pandas - matplotlib - 爬虫day01 - 1.什么是爬虫? - 就是通过编写程序模拟浏览器上网,然后让其去互联网上获取数据的过程 - 爬虫的分类: - 通用爬虫:爬取的是一整张页面源码数据 - 聚焦爬虫:爬取页面中局部的内容. - 关联:聚焦是建立在通用爬虫基础之上 - 增量式爬虫: - 用来检测网站数据更新的情况,从而爬取到网站中最新更新出来的数据. - 反爬机制:对应的载体是门户网站.网站中可以指定相关的机制防止爬虫程序对其网站中的数据进行爬取 - 反反爬策略:对应的载体是爬虫程序.爬虫程序可以破解网站采取的反爬机制,从而使得爬虫程序可以爬取到数据 - 第一个反爬机制:robots.txt协议 - User-Agent:请求载体的身份标识 - 特性:文本协议.防君子不防小人 - 2.requests模块 - 环境的安装:pip install requests - requests作用: - 用于模拟浏览器发起网络请求

easy_install lxml on Python 2.7 on Windows

由 孤者浪人 提交于 2019-11-30 06:19:24
问题 I'm using python 2.7 on Windows. How come the following error occurs when I try to install [lxml][1] using [setuptools][2]'s easy_install? C:\>easy_install lxml Searching for lxml Reading http://pypi.python.org/simple/lxml/ Reading http://codespeak.net/lxml Best match: lxml 2.3.3 Downloading http://lxml.de/files/lxml-2.3.3.tgz Processing lxml-2.3.3.tgz Running lxml-2.3.3\setup.py -q bdist_egg --dist-dir c:\users\my_user\appdata\local\temp\easy_install-mtrdj2\lxml-2.3.3\egg-dist-tmp-tq8rx4

Can I supply a URL to lxml.etree.parse on Python 3?

由 偶尔善良 提交于 2019-11-30 04:37:17
问题 The documentation says I can: lxml can parse from a local file, an HTTP URL or an FTP URL. It also auto-detects and reads gzip-compressed XML files (.gz). (from http://lxml.de/parsing.html under "Parsers") but a quick experiment seems to imply otherwise: Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> parser = etree.HTMLParser() >>> from urllib
订阅 lxml