How can I get all the plain text from a website with Scrapy?

Question

I would like to have all the text visible from a website, after the HTML is rendered. I\'m working in Python with Scrapy framework. With xpath(\'//body//text()\')

Answer 1

The xpath('//body//text()') doesn't always drive dipper into the nodes in your last used tag(in your case body.) If you type xpath('//body/node()/text()').extract() you will see the nodes which are in you html body. You can try xpath('//body/descendant::text()').