Beautiful Soup not waiting until page is fully loaded

Question

So with my code below I want to open an apartment website URL and scrape the webpage. The only issue is that Beautiful Soup isn\'t waiting until the entire webpage is render

Answer 1

If you want to wait for the page to fully load its data you should think about using selenium, in your case it could look like this:

from bs4 import BeautifulSoup
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

url = ""

chrome_options = Options()  
chrome_options.add_argument("--headless") # Opens the browser up in background

with Chrome(options=chrome_options) as browser:
     browser.get(url)
     html = browser.page_source

page_soup = BeautifulSoup(html, 'html.parser')
containers = page_soup.findAll("div",{"class":"grid-item"})