Click button on website then scrape web page

Question

I have a website I would like to click a button on then scrape the website using python the html code between the button is:

 <span id="exchange-testing" class="exchange-input nav-link" data track="&amp;lid=testing&amp;lpos=site_settings" data-value="testing">Testing</span>

Is this possible? I am able to scrape all the data I need from the page but I need to click the button first.

Any help would be appreciated

Answer 1

Basically, you have two options:

• high-level approach: automate a real browser using selenium or, in other words, make the browser repeat all the user actions needed to get to the page with the desired data.

• low-level approach: when you click the button, investigate what is happening under the hood - explore the "Network" tab of browser developer tools and see what requests are being made. Then, simulate them in your scraper. Here, you may consider using tools like requests, mechanize for making requests, handling scraping sessions, submitting forms etc and tools like BeautifulSoup, lxml.html for html parsing. Also, Scrapy web-scraping framework is a must see.