Adding a wait-for-element while performing a SplashRequest in python Scrapy

Question

I am trying to scrape a few dynamic websites using Splash for Scrapy in python. However, I see that Splash fails to wait for the complete page to load in certain cases. A br

Answer 1

I have a similar requirement, with timeouts. My solution is a slight modification of above:

function wait_css(splash, css, maxwait)
    if maxwait == nil then
        maxwait = 10     --default maxwait if not given
    end

    local i=0
    while not splash:select(css) do
       if i==maxwait then
           break     --times out at maxwait secs
       end
       i=i+1
       splash:wait(1)      --each loop has duration 1sec
    end
end