web-crawler

So my brother wanted me to write a web crawler in Python (self-taught) and I know C++, Java, and a bit of html. I'm using version 2.7 and reading the python library, but I have a few problems 1. httplib.HTTPConnection and request concept to me is new and I don't understand if it downloads an html script like cookie or an instance. If you do both of those, do you get the source for a website page? And what are some words that I would need to know to modify the page and return the modified page. Just for background, I need to download a page and replace any img with ones I have And it would be

I wanted to make a Java based web crawler for an experiment. I heard that making a Web Crawler in Java was the way to go if this is your first time. However, I have two important questions. How will my program 'visit' or 'connect' to web pages? Please give a brief explanation. (I understand the basics of the layers of abstraction from the hardware up to the software, here I am interested in the Java abstractions) What libraries should I use? I would assume I need a library for connecting to web pages, a library for HTTP/HTTPS protocol, and a library for HTML parsing. Mohammad Adil This is How

I've been using Scrapy web-scraping framework pretty extensively, but, recently I've discovered that there is another framework/system called pyspider , which, according to it's github page, is fresh, actively developed and popular. pyspider 's home page lists several things being supported out-of-the-box: Powerful WebUI with script editor, task monitor, project manager and result viewer Javascript pages supported! Task priority, retry, periodical and recrawl by age or marks in index page (like update time) Distributed architecture These are the things that Scrapy itself doesn't provide, but,

What is your recommendation of writing a web crawler in Ruby? Any lib better than mechanize? Nakilon If you want just to get pages' content, the simpliest way is to use open-uri functions. They don't require additional gems. You just have to require 'open-uri' and... http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html To parse content you can use Nokogiri or other gems, which also can have, for example, useful XPATH -technology. You can find other parsing libraries just here on SO . I'd give a try to anemone . It's simple to use, especially if you have to write a simple crawler.

Until recently there were several ways to retrieve Instagram user media without the need for API authentication. But apparently, the website stopped all of them. Some of the old methods: https://api.instagram.com/v1/users/user-id/media/recent/ https://www.instagram.com/user-id/media https://www.instagram.com/user-id/?__a=1 And some old related questions are: How can I get a user's media from Instagram without authenticating as a user? Is there still a way to fetch instagram feed without using access token now (06/2016)? I was able to retrieve the first twenty items by crawling the webpage of

I want to use the Python Scrapy module to scrape all the URLs from my website and write the list to a file. I looked in the examples but didn't see any simple example to do this. Here's the python program that worked for me: from scrapy.selector import HtmlXPathSelector from scrapy.spider import BaseSpider from scrapy.http import Request DOMAIN = 'example.com' URL = 'http://%s' % DOMAIN class MySpider(BaseSpider): name = DOMAIN allowed_domains = [DOMAIN] start_urls = [ URL ] def parse(self, response): hxs = HtmlXPathSelector(response) for url in hxs.select('//a/@href').extract(): if not ( url

I often have to work with fragile legacy websites that break in unexpected ways when logic or configuration are updated. I don't have the time or knowledge of the system needed to create a Selenium script. Besides, I don't want to check a specific use case - I want to verify every link and page on the site. I would like to create an automated system test that will spider through a site and check for broken links and crashes. Ideally, there would be a tool that I could use to achieve this. It should have as many as possible of the following features, in descending order of priority: Triggered