python - Reading multiple CSVs for Keras LSTM

问题

I'm trying to implement a LSTM network using Keras but I'm having problems with taking input. My dataset is in the form of multiple CSV files (all files have same dimensions 68x250 with each entry containing 2 values). There are about 200 CSV files, between various classes. Preview of one of the CSVs

How do i take these multiple CSVs as input?

回答1:

I did something similar recently, as Pedro said you shoudl use fit_generator and write your custom generator.

Here is an example of a generator:

def generator(files):
    print('start generator')
    while 1:        
        print('loop generator')
        for file in files:
            try:                 
                df = pd.read_csv(file)
                batches = int(np.ceil(len(df)/batch_size))      
                for i in range(0, batches):                                   
                    yield pad_batch(df[i*batch_size:min(len(df), i*batch_size+batch_size)])

            except EOFError:
                print("error" + file)

Where you pass the list of filename to the generator and it iterates through the files and returns the content in batches. load_data is in my case a function which reads csvs in pandas and does some preprocessing. pad_batch does the padding for the LSTM.

Usage:

model.fit_generator(
      generator=generator(trainingFiles),   
      steps_per_epoch=steps,
      epochs=num_epochs,
      validation_data=[x_test, y_test],
      verbose=1)

回答2:

Define a class that implements the interface defined in: https://keras.io/utils/#sequence

and use the method model.fit_generator.

来源：https://stackoverflow.com/questions/54879511/python-reading-multiple-csvs-for-keras-lstm

标签

python

csv

keras

lstm