How to “stop” and “resume” long time running Python script?

Question

I wrote Python script that processes big number of large text files and may run a lot of time. Sometimes, there is a need to stop the running script and to

Answer 1

If you are looking to read big files, just use a file handle, and read the lines one at a time, processing each line as you need to. If you'd like to save the python session, then just use dill.dump_session -- and it will save all existing objects. Other answers will fail as pickle cannot pickle a file handle. dill, however, can serialize almost every python object -- including a file handle.

Python 2.7.9 (default, Dec 11 2014, 01:21:43) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> f = open('bigfile1.dat', 'r')
>>> data = f.readline()  
>>> 
>>> dill.dump_session('session.pkl')
>>> 

Then quit the python session, and restart. When you load_session, you load all the objects that existed at the time of the dump_session call.

dude@hilbert>$ python
Python 2.7.9 (default, Dec 11 2014, 01:21:43) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('session.pkl')
>>> len(data)
9
>>> data += f.readline()
>>> f.close()
>>> 

Simple as that.

Get dill here: https://github.com/uqfoundation