beautifulsoup

问题 I am required to scrape PDF's from the website https://secc.gov.in/lgdStateList . There are 3 drop-down menus for a state, a district and a block. There are several states, under each state we have districts and under each district there are blocks. I tried to implement the following code. I was able to select the state, but there seems to be some error when I select the district. from selenium import webdriver from selenium.webdriver.support.ui import Select import requests from bs4 import

问题 I'm looking to scrape this link, with just two simple pieces of information, but I don't know why I have this result and it can't give me all the data I search for: particulier_allinfo particulier_tel 0 ABEL KEVIN10 RUE VIRGILE67200 Strasbourg This is the code, thanks for your help : import bs4 as bs import urllib import urllib.request import requests from bs4 import BeautifulSoup import pandas from pandas import DataFrame import csv with open('test_bs_118000.csv', mode='w') as csv_file:

问题 I want to scrape news from this website: https://www.bbc.com/news You can see that website has categories such as Home, US Election, Coronavirus etc. For example, If I go to specific news article such as: https://www.bbc.com/news/election-us-2020-54912611 I can write a scraper that will give me the headline, this is the code: from bs4 import BeautifulSoup response = requests.get("https://www.bbc.com/news/election-us-2020-54912611", headers=headers) soup = BeautifulSoup(response.content, 'html

问题 I have an xml which has multiple tags which look like this: <textblock height="55" hpos="143" id="Page1_Block5" lang="en-US" stylerefs="StyleId-E6BF91A3-3D6A-442F-9A46-22A0459A02E9- font1" vpos="226" width="393"> I want to get all the <textblock> tags clustered by a Page ( id property in the textblock tag). However, my id is written in the following way: id="Page1_Block5" . However, I want to condition only on the Page number, and not the block number. (I want all blocks of a specific page).

问题 So I came from the question here Now I am able to interact with the page, scroll down the page, close the popup that appears and click at the bottom to expand the page. The problem is when I count the items, the code only returns 20 and it should be 40. I have checked the code again and again - I'm missing something but I don't know what. See my code below: from selenium import webdriver from bs4 import BeautifulSoup import pandas as pd import time import datetime options = webdriver

问题 My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included). My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of

问题 My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included). My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of

问题 I have imported and modified some xml, but when I write out my xml using test.prettify(). It changes the top line of the xml from <?xml version="1.0"?> to <?xml version="1.0" encoding="utf-8"?> I don't want this change. How can I just keep the first line unchanged? What is the easiest way to do this? If it matters, I'm using the xml parser. soup = BeautifulSoup(r.text,'xml') 回答1: I'm sure there's a more elegant way to do this using BeautifulSoup's built-ins, but based on your comment, I'll

问题 I am trying to extract the text from a html file. The html file looks like this: <li class="toclevel-1 tocsection-1"> <a href="#Baden-Württemberg"><span class="tocnumber">1</span> <span class="toctext">Baden-Württemberg</span> </a> </li> <li class="toclevel-1 tocsection-2"> <a href="#Bayern"> <span class="tocnumber">2</span> <span class="toctext">Bayern</span> </a> </li> <li class="toclevel-1 tocsection-3"> <a href="#Berlin"> <span class="tocnumber">3</span> <span class="toctext">Berlin</span

问题 I am trying to extract the text from a html file. The html file looks like this: <li class="toclevel-1 tocsection-1"> <a href="#Baden-Württemberg"><span class="tocnumber">1</span> <span class="toctext">Baden-Württemberg</span> </a> </li> <li class="toclevel-1 tocsection-2"> <a href="#Bayern"> <span class="tocnumber">2</span> <span class="toctext">Bayern</span> </a> </li> <li class="toclevel-1 tocsection-3"> <a href="#Berlin"> <span class="tocnumber">3</span> <span class="toctext">Berlin</span