Scrapy + Splash: scraping element inside inner html

Question

I'm using Scrapy + Splash to crawl webpages and try to extract data form google ad banners and other ads and I'm having difficulty getting scrapy to follow the xpath into them.

I'm using the Scrpay-Splash API to render the pages so their scripts and images load and to take screenshots but it seems google ad banners are created by JS scripts that then insert their contents into a new html document within an iframe in the webpage, as so:

Splash makes sure the code is rendered so I don't run into the usual problem scrapy has with scripts where it reads the script's content instead of it's resulting html -- but I can't seem to find a way to indicate the XPath necessary to get to the element nodes I need (ad's href link).

If I inspect the element in google and copy it's xpath it simply gives me //*[@id="aw0"], which I feel would work if the iframe's html was all there was here, but it returns empty no matter how I write it and I fele it's probably because XPath doesn't elegantly handle html documents stacked within html documents.

The XPath to the iframe that contains the google ad code is //*[@id="google_ads_iframe_/87824813/hola/blogs/home_0"]{the numbers are constant}.

Is there a way to stack these XPaths together to get scrapy to follow the trail into the container I need? Or should I be parsing the Splash response object directly in some other way and I can't rely on Response.Xpath/Response.CSS for this?

Answer 1

The problem is that iframe content is not returned as a part of html. You can either try to fetch iframe content directly (by its src), or use render.json endpoint with iframes=1 option:

# ...
    yield SplashRequest(url, self.parse_result, endpoint='render.json', 
                        args={'html': 1, 'iframes': 1})

def parse_result(self, response):
    iframe_html = response.data['childFrames'][0]['html']
    sel = parsel.Selector(iframe_html)
    item = {
        'my_field': sel.xpath(...),
        # ...  
    }

/execute endpoint doesn't support fetching iframes content as of Splash 2.3.3.

Answer 2

An alternative way to deal with iframe can be (response if the main page):

    urls = response.css('iframe::attr(src)').extract()
    for url in urls :
            parse the url

this way the iframe is parsed like it was a normal page, but at the moment i cannot send the cookies in the main page to the html inside the iframe and that's a problem