How to web crawl some sites [closed]

问题

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed 5 years ago.

I am starting a new project of crawling websites to retrieve and store data internally using a web service. I looked up some information and came across Scrapy and Beevolve web crawling services.

My question is is it best to just create my own crawler with no prior experience or rent a web crawling service?

One issue that I came across is, some of the websites require a log in before getting any data.

回答1:

If you want to create your own web crawler in Java you may want to look at this

You could also take a look at jSpider and jsoup.

Edit : This could work too : crawler4j

来源：https://stackoverflow.com/questions/23917790/how-to-web-crawl-some-sites