Is there a way i can grab a column of a table that span over several pages using python?

问题

I am trying to get the tickers of ETFs from a table that spans over 46 pages:

http://etfdb.com/type/region/north-america/us/#etfs&sort_name=assets_under_management&sort_order=desc&page=1

My code is

import bs4 as bs
import pickle
import requests

def save_ETF_tickers():
    resp = requests.get('http://etfdb.com/type/region/north-america/us/#etfs&sort_name=assets_under_management&sort_order=desc&page=1')
    soup = bs.BeautifulSoup(resp.text, "lxml")
    table = soup.find('table',{'class': 'table mm-mobile-table table-module2 table-default table-striped table-hover table-pagination'})
    tickers = []
    for row in table.findAll('tr')[1:26]:
        ticker = row.findAll('td')[0].text
        tickers.append(ticker)

    with open("ETFtickers.pickle", "wb") as f:
        pickle.dump(tickers, f)

    print(tickers)

    return tickers

save_ETF_tickers()

I know that this one only check "page=1" but I couldn't figure out how to retrieve data from all of the 46 pages.

Your help would be very appreciated

回答1:

You could use the etfdb-api Node.js package: https://www.npmjs.com/package/etfdb-api

It provides you with:

Pagination
Sorting by: Year to Date Return, Price, AUM, Avg. Volume, etc.
Sort order: DESC | ASC

Here is a sample JSON response:

  {
    "symbol": {
      "type": "link",
      "text": "VIXM",
      "url": "/etf/VIXM/"
    },
    "name": {
      "type": "link",
      "text": "ProShares VIX Mid-Term Futures ETF",
      "url": "/etf/VIXM/"
    },
    "mobile_title": "VIXM - ProShares VIX Mid-Term Futures ETF",
    "price": "$26.47",
    "assets": "$48.21",
    "average_volume": "69,873",
    "ytd": "25.15%",
    "overall_rating": {
      "type": "restricted",
      "url": "/members/join/"
    },
    "asset_class": "Volatility"
  },
  {
    "symbol": {
      "type": "link",
      "text": "DGBP",
      "url": "/etf/DGBP/"
    },
    "name": {
      "type": "link",
      "text": "VelocityShares Daily 4x Long USD vs GBP ETN",
      "url": "/etf/DGBP/"
    },
    "mobile_title": "DGBP - VelocityShares Daily 4x Long USD vs GBP ETN",
    "price": "$30.62",
    "assets": "$4.85",
    "average_volume": "1,038",
    "ytd": "25.13%",
    "overall_rating": {
      "type": "restricted",
      "url": "/members/join/"
    },
    "asset_class": "Currency"
  }

Disclaimer: I'm the author. :)

来源：https://stackoverflow.com/questions/47690981/is-there-a-way-i-can-grab-a-column-of-a-table-that-span-over-several-pages-using

标签

python

beautifulsoup