I have a page that i need to get the source to use with BS4, but the middle of the page takes 1 second(maybe less) to load the content, and requests.get catches the source o
Just to list my way of doing it, maybe it can be of value for someone:
max_retries = # some int
retry_delay = # some int
n = 1
ready = 0
while n < max_retries:
try:
response = requests.get('https://github.com')
if response.ok:
ready = 1
break
except requests.exceptions.RequestException:
print("Website not availabe...")
n += 1
time.sleep(retry_delay)
if ready != 1
print("Problem")
else:
print("All good")