Python how to read N number of lines at a time

后端 未结 6 1104
感情败类
感情败类 2020-11-27 03:42

I am writing a code to take an enormous textfile (several GB) N lines at a time, process that batch, and move onto the next N lines until I have completed the entire file.

6条回答
  •  庸人自扰
    2020-11-27 04:09

    The question appears to presume that there is efficiency to be gained by reading an "enormous textfile" in blocks of N lines at a time. This adds an application layer of buffering over the already highly optimized stdio library, adds complexity, and probably buys you absolutely nothing.

    Thus:

    with open('my_very_large_text_file') as f:
        for line in f:
            process(line)
    

    is probably superior to any alternative in time, space, complexity and readability.

    See also Rob Pike's first two rules, Jackson's Two Rules, and PEP-20 The Zen of Python. If you really just wanted to play with islice you should have left out the large file stuff.

提交回复
热议问题