Python Scrapy Dynamic Web Sites

Question

I am trying to scrape a very simple web page with the help of Scrapy and it\'s xpath selectors but for some reason the selectors I have do not work in Scrapy but they do wor

Answer 1

I think I found the webpage you want to extract from, and the chapters are loaded after fetching some JSON data, based on a "mangaid" (that is available in a Javascript Array in the page.

So fetching the chapters is a matter of making a specific GET request to a specific /actions/selector/ endpoint. It's basically emulating what your browser's Javascript engine is doing.

You probably get better performance using this technique than Selenium, but it does involve (minor) Javascript parsing (no real interpretation needed).