Is it possible to parallelize selenium webdriver get_attribute calls in python?

Question

I am running this code

from multiprocessing.Pool import ThreadPool
from selenium import webdriver
driver = webdriver.Firefox()
driver.get(url)
elements = driver.find_elements_by_class_name("class-name")
pool = ThreadPool(4)
async = [pool.apply_async(fn_which_calls_get_attribute,(element,)) for element in elements]
results = [result.get() for result in async]

which works fine for some of the results, but throws an error of ResponseNotReady for other results. It runs as expected if I use "pool.apply" instead of the async version.

Is it a problem that I am making multiple calls to the selenium driver at once, and the error is because it cannot handle it? Or is something wrong with my parallelization?

Answer 1

Just Hint that Selenium run in a single thread and in a single core system. So its not possible to exercise multi-threading over selenium webdriver . Yes you can create a separate instance and attach to another core of a multi core system.

I may not answer your question but if you are trying to do something similar good not to do.