web-crawler

I'm trying to find a text change regularly inside javascript tag : <script type="text/javascript"> jwplayer("mediaplayer").setup({ flashplayer: "player.swf", file:"filename", provider: "rtmp", streamer:"rtmp://192.168.1.1/file?wmsAuthSign=RANDOM-114-Character==", height:500, width:500, }); </script> How to get RANDOM-114-Character (or full value of 'streamer' flashvars) using PHP Simple HTML DOM Parser, I just have no idea to do this. You can do it with regular expression: preg_match ($pattern, $java_script, $matches); Pattern depends, if the variable 'wmsAuthSign' is unique. For example:

My project structure: kmss/ ├── kmss │ ├── __init__.py │ ├── items.py │ ├── pipelines.py │ ├── settings.py │ └── spiders │ ├── __init__.py │ └── first.py ├── README.rst ├── scrapy.cfg └── setup.py I am running it on mac and my project folder is created at the location: /user/username/kmss And within items.py I do have a class named " KmssItem " . If I am going to run the first.py ( my spider), I have to import items.py. , which is at a higher level. I am having problem with the following line: from kmss.items import KmssItem Within items.py , the codes are: from scrapy import Item, Field class

I'm creating a small app for myself where I run a Ruby script and save all of the images off of my blog. I can't figure out how to save the image files after I've identified them. Any help would be much appreciated. require 'rubygems' require 'nokogiri' require 'open-uri' url = '[my blog url]' doc = Nokogiri::HTML(open(url)) doc.css("img").each do |item| #something end Phrogz URL = '[my blog url]' require 'nokogiri' # gem install nokogiri require 'open-uri' # already part of your ruby install Nokogiri::HTML(open(URL)).xpath("//img/@src").each do |src| uri = URI.join( URL, src ).to_s # make

I want to crawl an entire website , I have read several threads but I cannot manage to get data in a 2nd level. That is, I can return the links from a starting page but then I cannot find a way to parse the links and get the content of each link... The code I use is: <?php // SELECT STARTING PAGE $url = 'http://mydomain.com/'; $html= file_get_contents($url); // GET ALL THE LINKS OF EACH PAGE // create a dom object $dom = new DOMDocument(); @$dom->loadHTML($html); // run xpath for the dom $xPath = new DOMXPath($dom); // get links from starting page $elements = $xPath->query("//a/@href");

I'm tasked with crawling website built with React. I'm trying to fill in input fields and submitting the form using javascript injects to the page (either selenium or webview in mobile). This works like a charm on every other site + technology but React seems to be a real pain. so here is a sample code var email = document.getElementById( 'email' ); email.value = 'example@mail.com'; I the value changes on the DOM input element, but the React does not trigger the change event. I've been trying plethora of different ways to get the React to update the state. var event = new Event('change', {

I am crawling data using HtmlUnit from a dynamic webpage, which uses infinite scrolling to fetch data dynamically, just like facebook's newsfeed. I used the following sentence to simulate the scrolling down event: webclient.setJavaScriptEnabled(true); webclient.setAjaxController(new NicelyResynchronizingAjaxController()); ScriptResult sr=myHtmlPage.executeJavaScript("window.scrollBy(0,600)"); webclient.waitForBackgroundJavaScript(10000); myHtmlPage=(HtmlPage)sr.getNewPage(); But it seems myHtmlPage stays the same with the previous one, i.e., new data is not appended in myHtmlPage, as a result