Speed up writing to files

Question

I\'ve profiled some legacy code I\'ve inherited with cProfile. There were a bunch of changes I\'ve already made that have helped (like using simplejson\'s C extensions!).

Answer 1

Batching the writes into groups of 500 did indeed speed up the writes significantly. For this test case the writing rows individually took 21.051 seconds in I/O, while writing in batches of 117 took 5.685 seconds to write the same number of rows. Batches of 500 took a total of only 0.266 seconds.