error (429) Too Many Requests while geocoding with geopy in Python

Question

I have a Pandas dataframe with ~20k rows, and I am trying to geocode by address column into lat/long coordinates.

How do I use time.sleep() or maybe other function to stop OSM Nominatim from Too Many Requests 429 error that I am getting now?

Here's the code I use for this:

from geopy.geocoders import Nominatim
from geopy.distance import vincenty

geolocator = Nominatim()
df['coord'] = df['address'].apply(geolocator.geocode).apply(lambda x: (x.latitude, x.longitude))
df.head()

Thanks in advance!

Answer 1

geopy since 1.16.0 includes a RateLimiter class which provides a convenient way to deal with the Too Many Requests 429 error by adding delays between the queries and retrying the failed requests.

from geopy.geocoders import Nominatim
geolocator = Nominatim(user_agent="specify_your_app_name_here")

from geopy.extra.rate_limiter import RateLimiter
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)

df['coord'] = df['address'].apply(geocode).apply(lambda location: (location.latitude, location.longitude))
df.head()

Docs: https://geopy.readthedocs.io/en/1.16.0/#usage-with-pandas

Answer 2

I would imagine you use a for loop. Without seeing your data, it would look something like this.

x = df['address'].tolist()
names = []

for item in x:
    d={}
    a = geolocator.geocode(item, exactly_one=True, timeout=60)
    try:
        d["Latitude"] = a.latitude
    except:
        pass
    try:
        d["Longitude"] = a.longitude
    except:
        pass
    time.sleep(2)
    names.append(d)

d

This is how you would implement sleep to wait 2 seconds before running the loop again. Also, in the event that the geolocator cannot find the latitude and longitude, it will pass instead of exiting out of the loop and having you start over.