What is the difference between chain and chain.from_iterable in itertools?

Question

I could not find any valid example on the internet where I can see the difference between them and why to choose one over the other.

Answer 1

Extending @martijn-pieters answer

Although the access to the inner items in the iterable remains the same, and implementation wise,

• itertools_chain_from_iterable (i.e. chain.from_iterable in Python) and
• chain_new (i.e. chain in Python)

in the CPython implementation, are both duck-types of chain_new_internal

---

Are there any optimization benefits from using chain.from_iterable(x), where x is an iterable of iterable; and the main purpose is to ultimately consume the flatten list of items?

We can try benchmarking it with:

import random
from itertools import chain
from functools import wraps
from time import time

from tqdm import tqdm

def timing(f):
    @wraps(f)
    def wrap(*args, **kw):
        ts = time()
        result = f(*args, **kw)
        te = time()
        print('func:%r args:[%r, %r] took: %2.4f sec' % (f.__name__, args, kw, te-ts))
        return result
    return wrap

def generate_nm(m, n):
    # Creates m generators of m integers between range 0 to n.
    yield iter(random.sample(range(n), n) for _ in range(m))
    

def chain_star(x):
    # Stores an iterable that will unpack and flatten the list of list.
    chain_x = chain(*x)
    # Consumes the items in the flatten iterable.
    for i in chain_x:
        pass

def chain_from_iterable(x):
    # Stores an iterable that will unpack and flatten the list of list.
    chain_x = chain.from_iterable(x)
    # Consumes the items in the flatten iterable.
    for i in chain_x:
        pass


@timing
def versus(f, n, m):
  f(generate_nm(n, m))

P/S: Benchmark running... Waiting for the results.

---

Results

chain_star, m=1000, n=1000

for _ in range(10):
    versus(chain_star, 1000, 1000)

[out]:

func:'versus' args:[(, 1000, 1000), {}] took: 0.6494 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6603 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6367 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6350 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6296 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6399 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6341 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6381 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6343 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6309 sec

chain_from_iterable, m=1000, n=1000

for _ in range(10):
    versus(chain_from_iterable, 1000, 1000)

[out]:

func:'versus' args:[(, 1000, 1000), {}] took: 0.6416 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6315 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6535 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6334 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6327 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6471 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6426 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6287 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6353 sec
func:'versus' args:[(, 1000, 1000), {}] took: 0.6297 sec

chain_star, m=10000, n=1000

func:'versus' args:[(, 10000, 1000), {}] took: 6.2659 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2966 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2953 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.3141 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2802 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2799 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2848 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.3299 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2730 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.3052 sec

chain_from_iterable, m=10000, n=1000

func:'versus' args:[(, 10000, 1000), {}] took: 6.3129 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.3064 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.3071 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2660 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2837 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2877 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2756 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2939 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2715 sec
func:'versus' args:[(, 10000, 1000), {}] took: 6.2877 sec

chain_star, m=100000, n=1000

func:'versus' args:[(, 100000, 1000), {}] took: 62.7874 sec
func:'versus' args:[(, 100000, 1000), {}] took: 63.3744 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.5584 sec
func:'versus' args:[(, 100000, 1000), {}] took: 63.3745 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7982 sec
func:'versus' args:[(, 100000, 1000), {}] took: 63.4054 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.6769 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.6476 sec
func:'versus' args:[(, 100000, 1000), {}] took: 63.7397 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.8980 sec

chain_from_iterable, m=100000, n=1000

for _ in range(10):
    versus(chain_from_iterable, 100000, 1000)

[out]:

func:'versus' args:[(, 100000, 1000), {}] took: 62.7227 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7717 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7159 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7569 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7906 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.6211 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.7294 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.8260 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.8356 sec
func:'versus' args:[(, 100000, 1000), {}] took: 62.9738 sec

chain_star, m=500000, n=1000

for _ in range(3):
    versus(chain_from_iterable, 500000, 1000)

[out]:

func:'versus' args:[(, 500000, 1000), {}] took: 314.5671 sec
func:'versus' args:[(, 500000, 1000), {}] took: 313.9270 sec
func:'versus' args:[(, 500000, 1000), {}] took: 313.8992 sec

chain_from_iterable, m=500000, n=1000

for _ in range(3):
    versus(chain_from_iterable, 500000, 1000)

[out]:

func:'versus' args:[(, 500000, 1000), {}] took: 313.8301 sec
func:'versus' args:[(, 500000, 1000), {}] took: 313.8104 sec
func:'versus' args:[(, 500000, 1000), {}] took: 313.9440 sec