screen-scraping

问题 I am trying to obtain clinical images of psoriasis patients from these two websites for research purposes: http://www.dermis.net/dermisroot/en/31346/diagnose.htm http://dermatlas.med.jhmi.edu/derm/ For the first site, I tried just saving the page with firefox, but it only saved the thumbnails and not the full-sized images. I was able to access the full-sized images using a firefox addon called "downloadthemall", but it saved each image as part of a new html page and I do not know of any way

问题 The website I want to scrap is : http://www.mouthshut.com/mobile-operators/Reliance-Jio-reviews-925812061 I want to get the last page number of the above the link for proceeding, which is 499 while taking the screenshot. My code : from bs4 import BeautifulSoup from urllib.request import urlopen as uReq from selenium import webdriver;import time from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected

问题 I've been trying to scrap data from a WebSite, as my previous question indicates. I was able to figure what my problem was thanks to the comunity, but now I'm facing another problem. I don't get any error this time, but the program doesn't export any values to excel, my page still all blank. On the other website I was scraping from, the HTML.Elements were divs and now it's spans , it's because of that? Here's my code: Option Explicit Public Sub Loiça() Dim data As Object, i As Long, html As

问题 I am having a strange problem which seems to be a problem of ipv6 vs ipv4 dns names. I have a real time scraper which runs on my server which runs on ipv6 network. After scraping, this scraper returns some urls to images on a web page via ajax calls and then the images are shown in the browser on my local machine via the links returned by the scraper. But these urls are not resolved on my local network. My local machine does not run on ipv6 network. Also the web page being scraped hosts the

问题 >>require 'net/https' >>uri = URI('https://www.facebook.com/careers/department?dept=product-management&req=a2KA0000000E147MAC') >>conn = Net::HTTP.new(uri.host, uri.port) >>req = Net::HTTP::Get.new(uri.request_uri, {'User Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1'}) >>resp = conn.request req => #<Net::HTTPFound 302 Found readbody=true> The 302 redirection thrown by the website redirects to a 'unsupported

问题 I am trying work on a small data scraping stuff because I want to do some data analysis. For the data, I obtained from foxsports, the url link is also included in the codes. The steps are explained in the comment part. If possible, you could just paste and run. For the data, I want to jump over 2013-2018 seasons' web pages, and scrape all the data in the table on the web pages. So my codes are here: import requests from lxml import html import csv # Set up the urls for Bayern Muenchen's Team

问题 I'm trying to scrape the reviews from this link: https://www.google.com/search?q=google+reviews+2nd+chance+treatment+40th+street&rlz=1C1JZAP_enUS697US697&oq=google+reviews+2nd+chance+treatment+40th+street&aqs=chrome..69i57j69i64.6183j0j7&sourceid=chrome&ie=UTF-8#lrd=0x872b7179b68e33d5:0x24b5517d86a95f89,1 For what I'm using the following code to load the page from selenium import webdriver import datetime import time import argparse import os import time #Define the argument parser to read in

问题 I am using XPath to query HTML sites, which works pretty good so far, but now I hit a (brick)wall and can't find a solution :-) The html looks like this: <ul> <li><a href="">Text1<span>AnotherText1</span></a></li> <li><a href="">Text2<span>AnotherText2</span></a></li> <li><a href="">Text3<span>AnotherText3</span></a></li> </ul> I want to select the "TextX" part, but NOT the AnotherTextX part in the <span></span> So far I couldn't come up with any (pure) XPath solution to do that (and in my

问题 I have made a spider to get reviews from a page like this here using scrapy. I want product reviews only till a certain date(2nd July 2016 in this case). I want to close my spider as soon as the review date goes earlier than the given date and return the items list. Spider is working well but my problem is that i am not able to close my spider if the condition is met..if i raise an exception, spider closes without returning anything. Please suggest the best way to close the spider manually.

问题 Scraping article from New York Times site and getting an empty result set. My aim is to get the urls and the text of the h3 items. When I run this I get an empty set. Printing the section scrape shows I'm on the right path... target url - http://query.nytimes.com/search/sitesearch/?action=click&contentCollection&region=TopBar&WT.nav=searchWidget&module=SearchSubmit&pgtype=sectionfront#/san+diego/24hours url = "http://query.nytimes.com/search/sitesearch/?action=click&contentCollection&region