Iterate through excel files and sheets and concatenate in Python

问题

Say I have a folder which have multiple excel files with extension xlsx or xls, they share same header column a, b, c, d, e except some empty sheet in several files.

I want to iterate all the files and sheets (except for empty sheets) and concatenate them into one sheet of one file output.xlsx.

I have iterated through all excel files and append them to one file, but how could I iterate through all the sheets of each files if they have more than one sheets?

I need to integrate two block of code below into one. Thanks for your help.

import pandas as pd
import numpy as np
import glob

path = os.getcwd()
files = os.listdir(path)
files

df = pd.DataFrame()

# method 1

excel_files = [f for f in files if f[-4:] == 'xlsx' or f[-3:] == 'xls']
excel_files

for f in excel_files:
    data = pd.read_excel(f)
    df = df.append(data)

# method 2

for f in glob.glob("*.xlsx" or "*.xls"):
    data = pd.read_excel(f)
    df = df.append(data, ignore_index=True)

# save the data frame
writer = pd.ExcelWriter('output.xlsx')
df.to_excel(writer, 'sheet1')
writer.save()

For one file to concatenate multiple sheets:

file = pd.ExcelFile('file.xlsx')

names = file.sheet_names  # read all sheet names

df = pd.concat([file.parse(name) for name in names])

回答1:

import pandas as pd

path = os.getcwd()
files = os.listdir(path)
files

excel_files = [file for file in files if '.xls' in file]
excel_files

def create_df_from_excel(file_name):
    file = pd.ExcelFile(file_name)

    names = file.sheet_names

    return pd.concat([file.parse(name) for name in names])

df = pd.concat(
    [create_df_from_excel(xl) for xl in excel_files]
)

# save the data frame
writer = pd.ExcelWriter('output.xlsx')
df.to_excel(writer, 'sheet1')
writer.save()

来源：https://stackoverflow.com/questions/56033013/iterate-through-excel-files-and-sheets-and-concatenate-in-python

标签

python

pandas

dataframe

concatenation