Python sqlite3 and concurrency

Question

I have a Python program that uses the \"threading\" module. Once every second, my program starts a new thread that fetches some data from the web, and stores this data to my

Answer 1

I could not find any benchmarks in any of the above answers so I wrote a test to benchmark everything.

I tried 3 approaches

1. Reading and writing sequentially from the SQLite database
2. Using a ThreadPoolExecutor to read/write
3. Using a ProcessPoolExecutor to read/write

The results and takeaways from the benchmark are as follows

1. Sequential reads/sequential writes work the best
2. If you must process in parallel, use the ProcessPoolExecutor to read in parallel
3. Do not perform any writes either using the ThreadPoolExecutor or using the ProcessPoolExecutor as you will run into database locked errors and you will have to retry inserting the chunk again

You can find the code and complete solution for the benchmarks in my SO answer HERE Hope that helps!