How should I scrape these images without errors?

问题

I'm trying to scrape the images (or the images link) of this forum (http://www.xossip.com/showthread.php?t=1384077) . I've tried beautiful soup 4 and here is the code I tried:

import requests
from bs4 import BeautifulSoup

def spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.xossip.com/showthread.php?t=1384077&page=' + str(page)
        sourcecode= requests.get(url)
        plaintext = sourcecode.text
        soup = BeautifulSoup(plaintext)
        for link in soup.findAll('a',{'class': 'alt1'}):
            src = link.get('src')
            print(src)


        page += 1
spider(1)

How should I correct it so that I get links of images like pzy.be/example ?

回答1:

Okay, so I did this by getting all of the #post_message_* divs and then getting the images from each of those.

import requests
from bs4 import BeautifulSoup

def spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.xossip.com/showthread.php?t=1384077&page=' + str(page)
        sourcecode= requests.get(url)
        plaintext = sourcecode.text
        soup = BeautifulSoup(plaintext)
        divs = soup.findAll('div', id=lambda d: d and d.startswith('post_message_'))
        for div in divs:
            src = div.find('img')['src']
            if src.startswith('http'): # b/c it could be a smilie or something like that 
                print(src)

        page += 1

spider(1)

回答2:

The simplest way is to just request each page and filter the img tags:

from bs4 import BeautifulSoup
from requests import get
import re

def get_wp():
    start_url = "http://www.xossip.com/showthread.php?t=1384077&page={}"
    for i in range(73):
        r = get(start_url.format(i))
        soup = BeautifulSoup(r.content)
        for img in (i["src"] for i in  soup.find_all("img", src=re.compile("http://pzy.be.*.jpg"))):
           yield img

来源：https://stackoverflow.com/questions/35461772/how-should-i-scrape-these-images-without-errors