Scraping from javascript using Scrapy

问题

I need to scrape the content with javascript tag using scrapy as follows:

<script type='text/javascript' id='script-id'> attribute={"pid":"123","title":"abc","url":"http://example.com","date":"2014-07-31 14:56:39 CDT","channels":["test"],"tags":[],"authors":["james Catcher"]};</script>

I can extract the content using xpath

response.xpath('id("script-id")//text()').extract()

Output

[u'\nattribute = {"pid":"123","title":"abc","url":"http:/example.com","date":"2014-07-30 15:34:10 ","channels":["test"],"tags":[],"authors":["james Watt"]};\n(function( ){\n var s = document.createElement(\'script\');\n s.async = true;\n s.type = \'text/javascript\';\n s.src = document.location.protocol + \'//d8rk54i4mohrb. cloudfront.net/js/reach.js\';\n (document.getElementsByTagName(\'head\')[0] || document.getElementsByTagName(\'body\')[0]).appendChild(s);\n})();\n'']

How can I get each values using xpath?

回答1:

This is json, so you can first extract it from the string, then load it with json

In [1]: import json

In [2]: sample_string = [u'\n attribute={"pid":"123","title":"abc",'
        +'"url":"http:/example.com","date":"2014-07-30 15:34:10 ",'
        +'"channels":["test"],"tags":[],"authors":["james Watt"]}'][0]

In [3]: data = json.loads(sample_string[12:])

In [4]: data
Out[4]:
{u'authors': [u'james Watt'],
u'channels': [u'test'],
u'date': u'2014-07-30 15:34:10 ',
u'pid': u'123',
u'tags': [],
u'title': u'abc',
u'url': u'http:/example.com'}

In [5]: data['authors']
Out[5]: [u'james Watt']

Alternatively, you can also load a javascript engine like PyV8 to interpret those variables.

来源：https://stackoverflow.com/questions/25071994/scraping-from-javascript-using-scrapy

标签

python

xpath

scrapy