Extract value from javascript object in site using xpath and import.io

问题

I want extract a number provided by javascript object in site, but I really don't understand that I am doing.

I tried different versions using alike examples and guidelines in import.io site and other tutorial sites, but I got only 1 of two results: extracted all numbers on given page or nothing at all.

I tried e.g. //[contains(.,"Unikālo apmeklējumu skaits:")]@type ; //[contains(.,"Unikālo apmeklējumu skaits:")] . Most likely it's necessary to add there something else, but I just don't know that.

Link I am interested in to extract from is: https://www.ss.lv/msg/lv/clothes-footwear/womens-clothes/trousers/ikcbb.html and information necessary is a number after text "Unikālo apmeklējumu skaits:" which is given by javascript.

Hopefully someone will be able to help me with this problem.

回答1:

For someone who is new in web-scraping this should be a hard task, I'll ty to explain it. First of all, the xpath to get to that location could be something like this:

'//td[@class="msg_footer" and contains(text(), "Unik")]'

Now you have that tag (and what it contains), but if you check it doesn't contain the number you need, that content is being dynamically loaded with a javascript, and the javascript is this one:

<script type="text/javascript"><!-- 

var ss_w='rādīt numuru';
document.write( '<scr'+'ipt id="contacts_js" src="/js/2015-10-27/37863/VHoBGkpqSV8bfwkdTX9AXEpZXCVDlASIQ1ZV3kK.js?t='+new Date()+'"></scr'+'ipt>' );

--></script>

which could be gotten from the response with this xpath:

'//script[contains(text(), "contacts_js")]/text()'

from that string, you should replicate the url that comes in src, so this url for example:

/js/2015-10-27/37863/VHoBGkpqSV8bfwkdTX9AXEpZXCVDlASIQ1ZV3kK.js?t=

and add to the end the current date, as javascript creates it with new Date(). Then you should make a request to that url (adding the previous response domain), so something like:

https://www.ss.lv/js/2015-10-27/37863/VHoBGkpqSV8bfwkdTX9AXEpZXCVDlASIQ1ZV3kK.js?t=Wed%20Oct%2028%202015%2020:56:42%20GMT-0500%20(PET)

check that the date is urlencoded. it should return a response like:

var PHONE_CNT=-1;var PHONE_CNT2=-1;var PHONE_CNT3=-1;var EMAIL_CNT=-1;var SHOW_CNT=22;var PH_c="";var PH_1=0;var PH_2=0;var PH_3=0;

pcc_id=0;PH_1=gpzd("JTg3aCU3QyU1QnolN0MlN0JYcWh6JTVCdCU5NSU4QyU5MnV4ayU5QXElN0IlOTQlNUNweiU5MGtvJTdCJThFJTVF","55937369");

where you can check that the value inside SHOW_CNT is the number you want.

If you want to know how I figured out which request and which script was populating that response tag, well that I did using firebug, searching for SHOW_CNT inside all of the responses that involve calling to your URL, which pointed to the request I specified, and then trying to check who was requesting that.

Hope it helped.

回答2:

support@import.io are the guys to speak to, they give free advice and help trouble shoot problems just like this all the time.

There are all kinds of tips and tricks you can use... for example import.io provide (an undocumented beta) JavaScript Pre-render service that would likely work for you in this scenario. API publish failures are sometimes caused by timeouts while waiting for sites to render JS, this would fix that.

http://support.import.io/knowledgebase/articles/623235-infinite-scroll-and-javascript-prerender-beta

I hope this helps.

来源：https://stackoverflow.com/questions/33402951/extract-value-from-javascript-object-in-site-using-xpath-and-import-io

标签

xpath

import.io