Can I use a numpy array to generate folds for cross validation?

问题

I would like to use a numpy array to build folds for a k-folds cross validation task. Taking out the test slice is easy, but I can't figure out how to return the remainder of the array, with the test slice omitted. Is there an efficient way to do this?

examples = range(50)
classes = range(50)
data = np.array(zip(classes,examples))
test_slice = data[5:10]
train_on_remainder = ??

回答1:

You could set it up like so:

test_slice, remainder = np.split(data.copy(), [test_size], axis=0)
# run test
remainder[:test_size], test_slice = test_slice, remainder[:test_size].copy()
# run test
remainder[test_size:2*test_size], test_slice = test_slice, remainder[test_size:2*test_size].copy()

# etc.

I don't think you can have it with much less copying.

How it works:

.      full set:            | 0 | 1 | 2 | 3 | 4 | 5 |
       split (full copy)       / \
       tst / rem         | 0 |     | 1 | 2 | 3 | 4 | 5 |
         run trial
                             | 1 | 2 | 3 | 4 | 5 |
       swap tst and           ^ |
       first segment:         | v
       (partial copy)        | 0 |

       tst / rem         | 1 |     | 0 | 2 | 3 | 4 | 5 |
         run trial
                             | 0 | 2 | 3 | 4 | 5 |
       swap tst and               ^ |
       second segment:            | v
       (partial copy)            | 1 |

       tst / rem         | 2 |     | 0 | 1 | 3 | 4 | 5 |
         run trial
                             | 0 | 1 | 3 | 4 | 5 |
       swap tst and                   ^ |
       third segment:                 | v
       (partial copy)                | 2 |

etc. As you can see it is almost literally shifting the fold. Saving many full copies.

回答2:

sort of an odd question given that one normally would use sklearn's train_test_split() if its available.

edit: another approach might be

r = np.arange(len(data))
trainX = data[r < 5 | r > 10]

an efficient solution I'm not sure but try this build the indexers using a list comprehension.

def indx(n, test_slice):
    return [x for x in range(n) if, x not in test_slice]

test_slice = set(range(5, 10))
trainX = data[indx(len(data), test_slice))]

of course you should you something like sklearn's train_test_split() if its available.

回答3:

split = np.vsplit(data, np.array([5,10]))

'''This will give you a list with 3 elements'''

test_slice = split[1]
train_slice = np.vstack((split[0],split[2]))

[[5 5] [6 6] [7 7] [8 8] [9 9]]

[[ 0 0] [ 1 1] [ 2 2] [ 3 3] [ 4 4] [10 10] [11 11] [12 12] [13 13] [14 14] [15 15] [16 16] [17 17] [18 18] … [47 47] [48 48] [49 49]]

回答4:

Two approaches, demonstrated on a 1d array:

In [64]: data = np.arange(20)
In [65]: test = data[5:10]
In [66]: rest = np.concatenate((data[:5],data[10:]),axis=0)
In [67]: rest
Out[67]: array([ 0,  1,  2,  3,  4, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])
In [68]: 
In [68]: mask = np.zeros(data.shape[0], dtype=bool)
In [69]: mask[5:10] = True
In [70]: test = data[mask]
In [71]: rest = data[~mask]
In [72]: rest
Out[72]: array([ 0,  1,  2,  3,  4, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

There is a np.delete function

In [75]: np.delete(data, np.arange(5,10))
Out[75]: array([ 0,  1,  2,  3,  4, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

Internally it uses one of the two methods I demonstrated.

来源：https://stackoverflow.com/questions/42592584/can-i-use-a-numpy-array-to-generate-folds-for-cross-validation

标签

python

numpy

slice