I want to scrape contents from multiple tables in a webpage and the HTML code goes like this :
<div class="fixtures-table full-table-medium" id="fixtures-data">
<h2 class="table-header"> Date 1 </h2>
<table class="table-stats">
<tbody>
<tr class='preview' id='match-row-EFBO755307'>
<td class='details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team 1</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team 2</a>
</span>
</p>
</td>
</tr>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team 3</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team 4</a>
</span>
</p>
</td>
</tr>
</tbody>
</table>
<h2 class="table-header"> Date 2 </h2>
<table class="table-stats">
<tbody>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>team X</a>
</span>
<span class='team-away teams'>
<a href='random_team'>team Y</a>
</span>
</p>
</td>
</tr>
<tr class='preview' id='match-row-EFBO755307'>
<td class='match-details'>
<p>
<span class='team-home teams'>
<a href='random_team'>Team A</a>
</span>
<span class='team-away teams'>
<a href='random_team'>Team B</a>
</span>
</p>
</td>
</tr>
</tbody>
</table>
</div>
There are more matches under the dates (9 or 2 or 1 depending on the matches played on that date) and the no. of tables is 63 (which is equal to no. of days)
I want to extract, for each date, matches between teams and also which team is home and which team is away.
I was using the scrapy shell and tried following commands:
title = sel.xpath("//td[@class = 'match-details']")[0]
l_home = title.xpath("//span[@class = 'team-home teams']/a/text()").extract()
This printed a list of the home teams and this printed a list of all the away teams,
l_Away = title.xpath("//span[@class = 'team-away teams']/a/text()").extract()
This gave me a list for all the dates :
sel.xpath("/html/body/div[3]/div/div/div/div[4]/div[2]/div/h2/text()").extract()
What I want is for all dates get the matches that are played on a day (and also which team is home and away)
Should my items.py look like this:
date = Field()
home_team = Field()
away_team2 = Field()
Please help me to write the parse function and the Item class.
Thanks in advance.
Here's an example logic from scrapy shell:
>>> for table in response.xpath('//table[@class="table-stats"]'):
... date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
... print date
... for match in table.xpath('.//tr[@class="preview" and @id]'):
... home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
... away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]
... print home_team, away_team
...
Date 1
team 1 team 2
team 3 team 4
Date 2
team X team Y
Team A Team B
In the parse() method you would need to instantiate an Item instance in the inner loop and yield it:
def parse(self, response):
for table in response.xpath('//table[@class="table-stats"]'):
date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
for match in table.xpath('.//tr[@class="preview" and @id]'):
home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]
item = MyItem()
item['date'] = date
item['home_team'] = home_team
item['away_team'] = away_team
yield item
where Myitem would be:
class MyItem(Item):
date = Field()
home_team = Field()
away_team = Field()
来源：https://stackoverflow.com/questions/25064983/how-to-scrape-contents-from-multiple-tables-in-a-webpage