Web scraping program cannot find element which I can see in the browser

问题

I am trying to get the titles of the streams on https://www.twitch.tv/directory/game/Dota%202, using Requests and BeautifulSoup. I know that my search criteria are correct, yet my program does not find the elements I need.

Here is a screenshot showing the relevant part of the source code in the browser:

The HTML source as text:

<div class="tw-media-card-meta__title">
  <div class="tw-c-text-alt">
    <a class="tw-full-width tw-interactive tw-link tw-link--button tw-link--hover-underline-none tw-link--inherit" data-a-target="preview-card-title-link" href="/weplayesport_en">
      <div class="tw-align-items-start tw-flex">
        <h3 class="tw-ellipsis tw-font-size-5" title="NAVI vs HellRaisers | BO5 | ODPixel &amp; S4 | WeSave! Charity Play">NAVI vs HellRaisers | BO5 | ODPixel &amp; S4 | WeSave! Charity Play</h3>
      </div>
    </a>
  </div>
</div>

Here is my code:

import requests
from bs4 import BeautifulSoup

req = requests.get("https://www.twitch.tv/directory/game/Dota%202")

soup = BeautifulSoup(req.content, "lxml")

title_elems = soup.find_all("h3", attrs={"title": True})

print(title_elems)

When I run it, title_elems is just the empty list ([]).

Why is my program not finding the elements?

回答1:

The element you're interested in is dynamically generated, which means that your browser executed JavaScript, made other network requests, etc. in order to build the page. Requests is just an HTTP library, and as such will not do those things.

You could use a tool like Selenium, or perhaps even analyze the network traffic for the data you need and make the requests directly.

来源：https://stackoverflow.com/questions/60904786/web-scraping-program-cannot-find-element-which-i-can-see-in-the-browser