问题
I am trying to pull specific data out of the link provided below. When I run the code, it gives me all of the href links as expected, but when I try further testing for the same string, but using the contains syntax, it comes back as empty.
Ive checked read the docs, as well as DevHints, and everywhere I look, the "Contains" syntax is the recommended method to capture what Im looking for when all I know is that the syntax will be included, but not where or how.
Im trying to build a scraper to help a lot of people recently laid off find new work, so any assistance is greatly appreciated.
Code:
from lxml import html, etree
import requests
page = requests.get('https://ea.gr8people.com/index.gp?method=cappportal.showPortalSearch&sysLayoutID=123')
# print(page.content)
tree = html.fromstring(page.content)
print(tree)
# Select All Nodes
AllNodes = tree.xpath("//*")
# Select Only hyperlink nodes
AllHyperLinkNodes = tree.xpath("//*/a")
# Iterate through all Node Links
for node in AllHyperLinkNodes:
print(node.values())
print("======================================================================================================================")
# select using a condition 'contains'
# NodeThatContains = tree.xpath('//td[@class="search-results-column-left"]/text()')
NodeThatContains = tree.xpath('//*/a[contains(text(),"opportunityid")]')
for node in NodeThatContains:
print(node.values())
# Print the link that 'contains' the text
# print(NodeThatContains[0].values())
回答1:
BeautifulSoup based solution
from bs4 import BeautifulSoup
import requests
page = requests.get('https://ea.gr8people.com/index.gp?method=cappportal.showPortalSearch&sysLayoutID=123').content
soup = BeautifulSoup(page, 'html.parser')
links = soup.find_all('a')
links = [a for a in links if a.attrs.get('href') and 'opportunityid' in a.attrs.get('href')]
print('-- opportunities --')
for idx, link in enumerate(links):
print('{}) {}'.format(idx, link))
Output
-- opportunities --
0) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=154761&opportunityid=154761">
2D Capture Artist - 6 month contract
</a>
1) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=154426&opportunityid=154426">
Accounting Supervisor
</a>
2) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=152147&opportunityid=152147">
Advanced Analyst
</a>
3) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=153395&opportunityid=153395">
Advanced UX Researcher
</a>
4) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=151309&opportunityid=151309">
AI Engineer
</a>
5) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=150468&opportunityid=150468">
AI Scientist
</a>
6) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=151310&opportunityid=151310">
AI Scientist - NLP Focus
</a>
7) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=153351&opportunityid=153351">
AI Software Engineer (Apex Legends)
</a>
8) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=152737&opportunityid=152737">
AI Software Engineer (Frostbite)
</a>
9) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=154764&opportunityid=154764">
Analyste Qualité Sénior / Senior Quality Analyst
</a>
10) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=153948&opportunityid=153948">
Animator 1
</a>
11) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=151353&opportunityid=151353">
Applications Agreement Analyst
</a>
12) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=154668&opportunityid=154668">
AR Analyst I
</a>
13) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=153609&opportunityid=153609">
AR Specialist
</a>
14) <a href="index.gp?method=cappportal.showJob&layoutid=2092&inp1541=&inp1375=154773&opportunityid=154773">
Artiste Audio / Audio Artist
</a>
来源：https://stackoverflow.com/questions/55003947/using-lxml-with-html-requests-and-etree-it-gives-links-but-wont-let-me-searc