How to use tqdm with pandas in a jupyter notebook?

Question

I\'m doing some analysis with pandas in a jupyter notebook and since my apply function takes a long time I would like to see a progress bar. Through this post here I found t

Answer 1

I found that I had to import tqdm_notebook also. A simple example is given below that works in Jupyter notebook.

Given you want to map a function on a variable to create a new variable in your pandas dataframe.

# progress bar
from tqdm import tqdm, tqdm_notebook

# instantiate
tqdm.pandas(tqdm_notebook)

# replace map with progress_map
# where df is a pandas dataframe
df['new_variable'] = df['old_variable'].progress_map(some_function)

Answer 2

You can use:

tqdm_notebook().pandas(*args, **kwargs)

This is because tqdm_notebook has a delayer adapter, so it's necessary to instanciate it before accessing its methods (including class methods).

In the future (>v5.1), you should be able to use a more uniform API:

tqdm_pandas(tqdm_notebook, *args, **kwargs)

Answer 3

If you want to use more than 1 CPU for that slow apply step, consider using swifter. As a bonus, swifter automatically enables a tqdm progress bar on the apply step. To customize the bar description, use df.swifter.progress_bar(enable=True, desc='bar description').apply(...)

Answer 4

My working solution (copied form the documnetation):

from tqdm.auto import tqdm
tqdm.pandas()