Is there a more efficient way to load a large Mat object into memory than the FileStorage method in OpenCV?
I have a large Mat with 192 columns and 1 million rows I want to store locally in a file and load into memory then my application starts. There is no problem using the FileStorage, but I was wondering if there exists a more efficient method to do this. At the moment it takes about 5 minutes to load the Mat into memory using the Debug mode in Visual Studio and around 3 minutes in the Release mode and the size of the data file is around 1.2GB.
Is the FileStorage method the only method available to do this task?
Are you ok with a 100x speedup?
You should save and load your images in binary format. You can do that with the matwrite and matread function in the code below.
I tested both loading from a FileStorage and the binary file, and for a smaller image with 250K rows, 192 columns, type CV_8UC1 I got these results (time in ms):
// Mat: 250K rows, 192 cols, type CV_8UC1 Using FileStorage: 5523.45 Using Raw: 50.0879
On a image with 1M rows and 192 cols using the binary mode I got (time in ms):
// Mat: 1M rows, 192 cols, type CV_8UC1 Using FileStorage: (can't load, out of memory) Using Raw: 197.381
NOTE
- Never measure performance in debug.
- 3 minutes to load a matrix seems way too much, even for
FileStorages. However, you'll gain a lot switching to binary mode.
Here the code with the functions matwrite and matread, and the test:
#include #include #include using namespace std; using namespace cv; void matwrite(const string& filename, const Mat& mat) { ofstream fs(filename, fstream::binary); // Header int type = mat.type(); int channels = mat.channels(); fs.write((char*)&mat.rows, sizeof(int)); // rows fs.write((char*)&mat.cols, sizeof(int)); // cols fs.write((char*)&type, sizeof(int)); // type fs.write((char*)&channels, sizeof(int)); // channels // Data if (mat.isContinuous()) { fs.write(mat.ptr(0), (mat.dataend - mat.datastart)); } else { int rowsz = CV_ELEM_SIZE(type) * mat.cols; for (int r = 0; r (r), rowsz); } } } Mat matread(const string& filename) { ifstream fs(filename, fstream::binary); // Header int rows, cols, type, channels; fs.read((char*)&rows, sizeof(int)); // rows fs.read((char*)&cols, sizeof(int)); // cols fs.read((char*)&type, sizeof(int)); // type fs.read((char*)&channels, sizeof(int)); // channels // Data Mat mat(rows, cols, type); fs.read((char*)mat.data, CV_ELEM_SIZE(type) * rows * cols); return mat; } int main() { // Save the random generated data { Mat m(1024*256, 192, CV_8UC1); randu(m, 0, 1000); FileStorage fs("fs.yml", FileStorage::WRITE); fs > m1; double toc = (double(getTickCount()) - tic) * 1000. / getTickFrequency(); cout > dummy; return 0; }
