Is there a way to run multiple cells simultaneously in IPython notebook?

Question

One cell in my notebook executes for a long time, while the other CPU\'s in the machine are idle. Is it possible to run other cells in parallel?

Answer 1

Not magically, and probably not what you think but yes. Here is the documentation for ipyparallel (formerly IPython parallel) that will show you how to spawn multiple IPython kernel. After you are free to distribute the work across cores, and you can prefix cells with %%px0 %%px1... %%px999 (once set up) to execute a cell on a specific engine, which in practice correspond to parallel execution of cell. I woudl suggest having a look as Dask as well.

Answer 2

One possible workaround is that you may open 2 notebooks and copy one's result to the other after done.

Answer 3

This does not answer your question directly but I think it would help a lot of people that are having the same problem. You can move variables between notebooks easily and then continue running the functions on another notebook then move the result back to the main notebook.

For example:

Notebook 1:

%store X
%store y

Notebook 2:

%store -r X
%store -r y

new_df = ...
%store new_df

Notebook 1:

%store -r new_df