Frequently Updating Stored Data for a Numerical Experiment using Python [closed]

Answer 1

1. Assuming you are using numpy for your numerical experiments, instead of pickle I would suggest using numpy.savez.
2. Keep it simple and make optimizations only if it you feel that the script runs too long.
3. Opening and closing files does affect the run time, but having a backup is anyway better.

And I would use collections.defaultdict(list) instead of plain dict and setdefault.