BeautifulSoup returning none when element definitely exists

Question

I\'m new to web scraping and have been using BeautifulSoup to scrape daily mortgage rates. However, a lot of the servicer sites that I am trying to scrape return \'none\' or

Answer 1

To get the data you are after you can use selenium in combination with python something like below:

from bs4 import BeautifulSoup
from selenium  import webdriver

driver = webdriver.Chrome()
driver.get('https://www.popular.com/en/mortgages/')

soup = BeautifulSoup(driver.page_source,"lxml")
item = soup.select('.table-responsive')[0].select("span div")[0].text
print(item)
driver.quit()

Result:

2.75%