Beautiful Soup not waiting until page is fully loaded

Question

So with my code below I want to open an apartment website URL and scrape the webpage. The only issue is that Beautiful Soup isn\'t waiting until the entire webpage is render

Answer 1

I'm happy with requests_html library. It will render Dynamic HTML for you. And is much simpler to implement than Selenium.

from requests_html import HTMLSession
import pyppdf.patch_pyppeteer
from bs4 import BeautifulSoup

url = 'https://xxxxx.com/properties/?sort=latest'

session = HTMLSession()


resp = session.get(link)
resp.html.render()
html = resp.html.html

page_soup = BeautifulSoup(html, 'html.parser')

containers = page_soup.find_all("div", {"class": "grid-item"})