cPickle very large amount of data
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效,请关闭广告屏蔽插件后再试): 问题: I have about 0.8 million images of 256x256 in RGB, which amount to over 7GB. I want to use them as training data in a Convolutional Neural Network, and want to put them in a cPickle file, along with their labels. Now, this is taking a lot of memory, to the extent that it needs to swap with my hard drive memory, and almost consume it all. Is this is a bad idea? What would be the smarter/more practical way to load into CNN or pickle them without causing too much memory issue? This is what the code looks like import numpy as np import cPickle