Python combining all csv files in a directory and order by date time

问题

I have 2 years worth of daily data split into monthly files. I would like to combine all of this data into one file ordered by date and time. The code I am using combines all the files, but not in order.

Code I am using

import pandas as pd
import glob, os
import csv

inputdirectory = input('Enter the directory: ')
df_list = []

for filename in sorted(glob.glob(os.path.join(inputdirectory,"*.csv*"))):
    df_list.append(pd.read_csv(filename))
    full_df = pd.concat(df_list)
    full_df.to_csv('totalsum.csv', index=False)

回答1:

Pre-process the file list to sort it:

Create a list of file_names,
extract the relevant info from the name and create a datetime object,
sort on the datetime object,
then use the sorted list.

import operator
fyles = ['CB02 May 2014.dailysum',
         'CB01 Apr 2015.dailysum',
         'CB01 Jul 2015.dailysum',
         'CB01 May 2015.dailysum',
         'CB01 Sep 2015.dailysum',
         'CB01 Oct 2015.dailysum',
         'CB13 May 2015.dailysum',
         'CB01 Jun 2017.dailysum',
         'CB01 Aug 2015.dailysum'
         ]

new_fyles = []
for entry in fyles:
    day, month, year = entry.split()
    year, _ = year.split('.')
    day = day[-2:]
##    print(entry, (month, year))
    dt = datetime.datetime.strptime(' '.join((day, month, year)), '%d %b %Y')
##    print(entry, dt)
    new_fyles.append((entry, dt))

date = operator.itemgetter(1)
f_name = operator.itemgetter(0)
new_fyles.sort(key = date)
for entry in new_fyles:
    print(f_name(entry))

You can make the file list like this:

import os, os.path
fyles = [fn for fn in os.listdir(inputdirectory) if fn.endswith('.dailysum')]

Then, after sorting, write the contents of each file to the new file:

with open('totalsum.csv', 'w') as out:
    for entry in new_fyles:
        f_path = os.path.join(inputdirectory, f_name(entry))
        with open(f_path) as f:
            out.write(f.read())

You could perform the sorting in a function

date = operator.itemgetter(1)
f_name = operator.itemgetter(0)
def f_name_sort(f_list):
    '''Return sorted list of file names'''
    new_fyles = []
    for entry in f_list:
        day, month, year = entry.split()
        year, _ = year.split('.')
        day = day[-2:]
        dt = datetime.datetime.strptime(' '.join((day, month, year)), '%d %b %Y')
        new_fyles.append((entry, dt))
    new_fyles.sort(key = date)
    return [f_name(entry) for f_name in new_fyles]

and use it like this:

for entry in f_name_sort(fyles):
    ...

Or write a function that converts a filename to a datetime object and use it as the key for sorting

def key(f_name):
    day, month, year = f_name.split()
    year, _ = year.split('.')
    day = day[-2:]
    return datetime.datetime.strptime(' '.join((day, month, year)), '%d %b %Y')

fyles.sort(key = key)
for entry in fyles:
    ...

回答2:

after this line:

full_df = pd.concat(df_list)

you need to convert your column 'datecolumn' to a datetime column:

full_df['datecolumn'] = full_df['datecolumn'].to_datetime(format=r'%d/%m/%y')

(judging by your comments, that format should work)

finally you can use

full_df.sort_values(by='datecolumn').to_csv('totalsum.csv', index=False)

to sort and write it

来源：https://stackoverflow.com/questions/42979891/python-combining-all-csv-files-in-a-directory-and-order-by-date-time

标签

python

python-3.x

pandas