Scraping AJAX e-commerce site using python

Question

I have a problem on scraping an e-commerce site using BeautifulSoup. I did some Googling but I still can\'t solve the problem.

Please refer on the

Answer 1

Welcome to StackOverflow! :)

As an alternative, you can check Selenium

See example usage from documentation:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.clear()
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

When you use requests (or libraries like Scrapy) usually JavaScript not loaded. As @dmitrybelyakov mentioned you can reply these calls or imitate normal user interaction using Selenium.