web-crawler

I can't find any hint on how to setup nutch to NOT filter/remove my URL parameters. I want to crawl and index some pages where lots of content is hidden behind the same base URLs (like /news.jsp?id=1 /news.jsp?id=2 /news.jsp?id=3 and so on). the regex-normalize.xml only removes redundant stuff from the URL (like session id, and trailing ?) the regex-urlfilter.txt seems to have a wildcard for my host (+^http://$myHost/) The crawling works fine so far. Any ideas? cheers, mana EDIT: A part of the solution is hidden here: configuring nutch regex-normalize.xml # skip URLs containing certain

Is it possible to crawl check-in data from foursquare in a greedy way? (even if I don't have friendship with all the users) Just like crawling publicly available twitter messages. If you have any experience or suggestions, please share. Thanks. If you have publicly available tweets containing links to foursquare, you can resolve the foursquare short links (4sq.com/XXXXXX) by making a HEAD request. The head request will return a URL with a check-in ID and a signature. You can use those two values to retrieve a check-in object via the foursquare API /checkins/ endpoint. You're only allowed to

In Python 3 , the following code obtains the HTML source for a webpage. import urllib.request url = "https://docs.python.org/3.4/howto/urllib2.html" response = urllib.request.urlopen(url) response.read() How can I add the following custom header to the request when using urllib.request? headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' } The request headers can be customized by first creating a request object then supplying it to urlopen. import urllib.request url = "https://docs.python.org/3.4/howto/urllib2.html" hdr = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64;

I am trying to program a crawl spider to crawl RSS feeds of a website and then parsing the meta tags of the article. The first RSS page is a page that displays the RSS categories. I managed to extract the link because the tag is in a tag. It looks like this: <tr> <td class="xmlLink"> <a href="http://feeds.example.com/subject1">subject1</a> </td> </tr> <tr> <td class="xmlLink"> <a href="http://feeds.example.com/subject2">subject2</a> </td> </tr> Once you click that link it brings you the the articles for that RSS category that looks like this: <li class="regularitem"> <h4 class="itemtitle"> <a

Closed. This question is off-topic. It is not currently accepting answers. Learn more . Want to improve this question? Update the question so it's on-topic for Stack Overflow. I have a crawler that crawl a few different domains for new posts/content. The total amount of content is hundred of thousands of pages, and there is a lot of new content added each day. So to be able to crawl through all this content, I need my crawler to be crawling 24/7. Currently I host the crawler script on the same server as the site the crawler is adding the content to, and I'm only able to run a cronjob to run

When I try to take some nonexistent content from page I catch this error: The current node list is empty. 500 Internal Server Error - InvalidArgumentException How can I safely check exists this content or not? Here some examples that does not work: if($crawler->filter('.PropertyBody')->eq(2)->text()){ // bla bla } if(!empty($crawler->filter('.PropertyBody')->eq(2)->text())){ // bla bla } if(($crawler->filter('.PropertyBody')->eq(2)->text()) != null){ // bla bla } THANKS, I helped myself with: $count = $crawler->filter('.PropertyBody')->count(); if($count > 2){ $marks = $crawler->filter('