Wait page to load before getting data with requests.get in python 3

后端 未结 4 891
陌清茗
陌清茗 2020-11-28 10:40

I have a page that i need to get the source to use with BS4, but the middle of the page takes 1 second(maybe less) to load the content, and requests.get catches the source o

4条回答
  •  慢半拍i
    慢半拍i (楼主)
    2020-11-28 11:21

    Just to list my way of doing it, maybe it can be of value for someone:

    max_retries = # some int
    retry_delay = # some int
    n = 1
    ready = 0
    while n < max_retries:
      try:
         response = requests.get('https://github.com')
         if response.ok:
            ready = 1
            break
      except requests.exceptions.RequestException:
         print("Website not availabe...")
      n += 1
      time.sleep(retry_delay)
    
    if ready != 1
      print("Problem")
    else:
      print("All good")
    

提交回复
热议问题