how to execute scrapy shell “URL” with notebook

Question

i am trying to scrapy

and there is scrapy shell "URL" command, executing console ipython with a response object from the URL.

but i want to do that thing with notebook.

is there any way to execute scrapy shell with notebook,

or how can i get the same response object of the URL in ipython manually?

Answer 1

Stumbled across this unanswered question looking for the same answer. The closest thing I can come up right now is this:

import requests
from scrapy.http import TextResponse

r = requests.get('http://stackoverflow.com/')
response = TextResponse(r.url, body=r.text, encoding='utf-8')

Then you can debug response using usual methods like xpath().