Concatenate multiple files into a single file object without creating a new file

Question

This question is related to Python concatenate text files

I have a list of file_names, like [\'file1.txt\', \'file2.txt\', ...].

I wou

Answer 1

EDIT:

As pointed out in the comments, this solution probably would not be optimal for large files as it loads everything into memory. A solution using generators would be better if large files are involved. Thanks to LohmarASHAR for pointing that out!

You could just create on big list from all of the files by looping over the list of filenames. This wouldn't create a new file object, just a new list object:

filenames = ["f1.txt", "f2.txt", "f3.txt"]

# list to store each line of the file
output = []

# iterate over list of filenames
for text_file in filenames:

    # open file
    with open(text_file) as f:

        # iterate over each line in the file and add to output
        for line in f.readlines():
            output.append(line)

Not that we are not explicitly calling close() on the file, this is because the with... statement will close the file for us as soon as it goes out of scope.

If you are able to use external libraries, pandas might be worth looking into for storing the file data in efficient, easy-to-use objects.