web-scraping

问题 I am scraping a website with urls http://domain.com/post/X , where X is a number stating from 1:5000 I can scrap using rvest using this code: website <- html("http://www.domain.com/post/1") Name <- website%>% html_node("body > div > div.row-fluid > div > div.DrFullDetails > div.MainDetails > div.Description > h1") %>% html_text() Speciality <- website %>% html_node("body > div > div.row-fluid > div > div.DrFullDetails > div.MainDetails > div.Description > p.JobTitle") %>% html_text() I need

问题 I have a very simple problem. I'm trying to get the job description from the html of a linkedIn page, but instead of getting the html of the page I'm getting few lines that look like a javascript code instead. I'm very new to this so any help will be greatly appreciated! Thanks Here's my code: import requests url = "https://www.linkedin.com/jobs/view/inside-sales-manager-at-stericycle-1089095836/" page_html = requests.get(url).text print(page_html) When I run this I don't get the html that I

问题 I have a very simple problem. I'm trying to get the job description from the html of a linkedIn page, but instead of getting the html of the page I'm getting few lines that look like a javascript code instead. I'm very new to this so any help will be greatly appreciated! Thanks Here's my code: import requests url = "https://www.linkedin.com/jobs/view/inside-sales-manager-at-stericycle-1089095836/" page_html = requests.get(url).text print(page_html) When I run this I don't get the html that I

问题 I'm getting a list of 30 items from the code: const boxes = await page.evaluate(() => { return document.querySelectorAll("DIV.a-row.dealContainer.dealTile") }) console.log(boxes); The result { '0': {}, '1': {}, '2': {}, .... '28': {}, '29': {} } I have the need to see the html of the elements. But every property I tried of boxes is simply undefined . I tried length , innerHTML , 'innerText` and some other. I am sure of box really containing something because puppeteer's screenshot shows the

问题 I'm getting a list of 30 items from the code: const boxes = await page.evaluate(() => { return document.querySelectorAll("DIV.a-row.dealContainer.dealTile") }) console.log(boxes); The result { '0': {}, '1': {}, '2': {}, .... '28': {}, '29': {} } I have the need to see the html of the elements. But every property I tried of boxes is simply undefined . I tried length , innerHTML , 'innerText` and some other. I am sure of box really containing something because puppeteer's screenshot shows the

问题 I found this link: https://search.roblox.com/catalog/json?Category=2&Subcategory=2&SortType=4&Direction=2 The original is: https://www.roblox.com/catalog/?Category=2&Subcategory=2&SortType=4 I am trying to scrape the prices of all the items in the whole catalog with Python, but I can't seem to locate the prices of the items. The URL does not change whenever I go to the next page. I have tried inspecting the website itself but I can't manage to find anything. The first URL is somehow

问题 I have made a web scraper in python to give me information on when free bet offers from various bookie websites have changed or new ones have been added. However, the bookies tend to record information relating to IP traffic and MAC addresses in order to flag up matched betters. How can I spoof my IP address when using the Request() method in the urllib.request module? My code is below: req = Request('https://www.888sport.com/online-sports-betting-promotions/', headers={'User-Agent': 'Mozilla

问题 I have made a web scraper in python to give me information on when free bet offers from various bookie websites have changed or new ones have been added. However, the bookies tend to record information relating to IP traffic and MAC addresses in order to flag up matched betters. How can I spoof my IP address when using the Request() method in the urllib.request module? My code is below: req = Request('https://www.888sport.com/online-sports-betting-promotions/', headers={'User-Agent': 'Mozilla

问题 I'm actually trying to scrape some car datas from different websites, i've been using selenium with chromebrowser but some websites actually block selenium with captcha validation(example: https://www.leboncoin.fr/), and this in just 1 or 2 requests. I tried changing $_cdc in the chromebrowser but this didn't resolve the problem, and I've been using those options for the chromebrowser user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97

问题 I'm actually trying to scrape some car datas from different websites, i've been using selenium with chromebrowser but some websites actually block selenium with captcha validation(example: https://www.leboncoin.fr/), and this in just 1 or 2 requests. I tried changing $_cdc in the chromebrowser but this didn't resolve the problem, and I've been using those options for the chromebrowser user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97