web-crawler

问题 I'm facing a problem that I unable to get all the product data as the website using a lazy load on product catalog page. meaning it needs to scroll until the whole page loaded. I getting only first-page products data. 回答1: First, you should keep in mind that there are infinite ways that infinite scroll can be implemented. Sometimes you have to click buttons on the way or do any sort of transitions. I will cover only the most simple use-case here which is scrolling down with some interval and

问题 I'm facing a problem that I unable to get all the product data as the website using a lazy load on product catalog page. meaning it needs to scroll until the whole page loaded. I getting only first-page products data. 回答1: First, you should keep in mind that there are infinite ways that infinite scroll can be implemented. Sometimes you have to click buttons on the way or do any sort of transitions. I will cover only the most simple use-case here which is scrolling down with some interval and

问题 I'm facing a problem that I unable to get all the product data as the website using a lazy load on product catalog page. meaning it needs to scroll until the whole page loaded. I getting only first-page products data. 回答1: First, you should keep in mind that there are infinite ways that infinite scroll can be implemented. Sometimes you have to click buttons on the way or do any sort of transitions. I will cover only the most simple use-case here which is scrolling down with some interval and

问题 I'm facing a problem that I unable to get all the product data as the website using a lazy load on product catalog page. meaning it needs to scroll until the whole page loaded. I getting only first-page products data. 回答1: First, you should keep in mind that there are infinite ways that infinite scroll can be implemented. Sometimes you have to click buttons on the way or do any sort of transitions. I will cover only the most simple use-case here which is scrolling down with some interval and

问题 I am trying to write a crawler to automatically download some files using python requests module. However, I met a problem. I initialized a new requests session, then I used post method to login into the website, after that as long as I try to use post/get method (a simplified code below): s=requests.session() s.post(url,data=post_data, headers=headers) #up to here everything is correct, the next step will report error s.get(url) or s.post(url) even repeat s.post(url,data=post_data, headers

问题 I am trying to write a crawler to automatically download some files using python requests module. However, I met a problem. I initialized a new requests session, then I used post method to login into the website, after that as long as I try to use post/get method (a simplified code below): s=requests.session() s.post(url,data=post_data, headers=headers) #up to here everything is correct, the next step will report error s.get(url) or s.post(url) even repeat s.post(url,data=post_data, headers

问题 Hello everyone I'm trying to fetch content inside of script tag. http://www.teknosa.com/urunler/145051447/samsung-hm1500-bluetooth-kulaklik this is the website. Also this is script tag which I want to enter inside. $.Teknosa.ProductDetail = {"ProductComputedIndex":145051447,"ProductName":"SAMSUNG HM1500 BLUETOOTH KULAKLIK","ProductSeoName":"samsung-hm1500-bluetooth-kulaklik","ProductBarcode":"8808993790425","ProductPriceInclTax":79.9,"ProductDiscountedPriceInclTax":null,"ProductStockQuantity"

问题 I have a javascript on the page like below: new Shopify.OptionSelectors("product-select", { product: {"id":185310341,"title":"10. Design | Siyah \u0026 beyaz kalpli", i want to get "185310341". I am searching on google about a few hours but couldn't find anything, I hope u can help me. How can i scrape that javascript and get that id? I tried that code : id = sel.search('"id":(.*?),',text).group(1) print id but i got: exceptions.AttributeError: 'Selector' object has no attribute 'search' 回答1:

问题 I am working on a crawler based on the StormCrawler project. I have a requirement to crawl URLs based on their priorities. For example, I have two types of priority: HIGH, LOW. I want to crawl HIGH priority URLs as soon as possible before LOW URLs. I need a method for handling the above problem in the crawler. How can I handle this requirement in Apache Storm and StormCrawler? 回答1: With Elasticsearch as a backend, you can configure the spouts to sort the URLs within a bucket by whichever

问题 I am new to python and just downloaded it today. I am using it to work on a web spider, so to test it out and make sure everything was working, I downloaded a sample code. Unfortunately, it does not work and gives me the error: "AttributeError: 'MyShell' object has no attribute 'loaded' " I am not sure if the code its self has an error or I failed to do something correctly when installing python. Is there anything you have to do when installing python like adding environmental variables, etc.