How to use python to get google news headlines and search keywords?

问题

I am working on a project to look through google news headlines and find keywords.

I want it to: -put the headlines into a text file -remove commas, apostrophes, quotes, punctuation, etc -search keywords

This is the code I have so far. I am getting the headlines, I now just need it to parse the keywords from each individual headline.

from lxml import html
import requests

# Send request to get the web page
response = requests.get('http://news.google.com')

# Check if the request succeeded (response code 200)
if (response.status_code == 200):

# Parse the html from the webpage
pagehtml = html.fromstring(response.text)

# search for news headlines
news = pagehtml.xpath('//h2[@class="esc-lead-article-title"] \
                      /a/span[@class="titletext"]/text()')

# Print each news item in a new line
print("\n".join(news))

回答1:

Alright I fixed it.

from lxml import html
import requests

# Send request to get the web page
response = requests.get('https://news.google.com/news/section?cf=all&pz=1&topic=b&siidp=b458d5455b7379bd8193a061024cd11baa97&ict=ln')

# Check if the request succeeded (response code 200)
if (response.status_code == 200):

    # Parse the html from the webpage
    pagehtml = html.fromstring(response.text)

    # search for news headlines
    news = pagehtml.xpath('//h2[@class="esc-lead-article-title"] \
                          /a/span[@class="titletext"]/text()')

    # Print each news item in a new line
    print("\n \n".join(news))

tf = open("headlines.txt", "w")

tf.write("\n \n".join(news).lower())

tf.close()
# puts as lower case in text file named headlines

with open('headlines.txt', 'r') as inF:
    for line in inF:
        if 'inflation' in line:
             print "\n" + "    " + line
# searches for 'inflation' (or whatever query) and prints in indented on a new line

来源：https://stackoverflow.com/questions/41472367/how-to-use-python-to-get-google-news-headlines-and-search-keywords