Selenium : Unable to put scraped elements to csv

问题

I have successfully scraped the data from the site. Well it's returning me error.

I used "Title1" : pd.Series([ ele for ele.text in elements ]) for storing data to csv file, but returns me error that name "ele" not defined when i use element to .text method .

When i remove .text, then it runs fine. But stores the id's which are not in form of text, so that's why i used .text. What is happening with usage of .text?

Here is my code :

element = WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, x))
)
elements = driver.find_elements_by_css_selector(x)

element = WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, y))
)
elements2 = driver.find_elements_by_css_selector(y)

element = WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, z))
)
elements3 = driver.find_elements_by_css_selector(z)

df = pd.DataFrame({
    "Title1" : pd.Series([ ele for ele.text in elements ]),
    "Title2" : pd.Series([ ele2 for ele2.text in elements2 ]),
    "Title3" : pd.Series([ ele3 for ele3.text in elements3 ]),
})

df.to_csv(csv_file_location,
          index=False, mode='a', encoding='utf-8')

Just remove the text and see that it works fine and stores all the data to csv but not as text. Any help would be appreciated...

回答1:

This method .text() is simply used to fetch an array/list of elements. In this line for example,

elements = driver.find_elements_by_css_selector(x)

This is why your loop pd.Series([ ele for ele.text in elements ]), fails, and so removing the text runs fine as expected.

So change this

pd.Series([ ele for ele.text in elements ])

to this

pd.Series([ ele.text for ele in elements ])

This means that it would first obtain ele first in elements and within that obtain the text attribute of the ele.

来源：https://stackoverflow.com/questions/61113227/selenium-unable-to-put-scraped-elements-to-csv