How can I save a very large MATLAB sparse matrix to a text file?

Question

I have a 30000x14000 sparse matrix in MATLAB (version 7), which I need to use in another program. Calling save won't write this as ASCII (not supported). Calling full() on this monster results in an Out of Memory error.
How do I export it?

Answer 1

You can use find to get index & value vectors:

[i,j,val] = find(data)
data_dump = [i,j,val]

You can recreate data from data_dump with spconvert, which is meant to "Import from sparse matrix external format" (so I guess it's a good export format):

data = spconvert( data_dump )

You can save to ascii with:

save -ascii data.txt data_dump

But this dumps indices as double, you can write it out more nicely with fopen/fprintf/fclose:

fid = fopen('data.txt','w')
fprintf( fid,'%d %d %f\n', transpose(data_dump) )
fclose(fid)

Hope this helps.

Answer 2

Save the sparse matrix as a .mat file. Then, in the other program, use a suitable library to read the .mat file.

For instance, if the other program is written in Python, you can use the scipy.io.mio.loadmat function, which supports sparse arrays and gives you a sparse numpy matrix.

Answer 3

I saved it as text using Java within MATLAB. MATLAB Code:

pw=java.io.PrintWriter(java.io.FileWriter('c:\\retail.txt'));
line=num2str(0:size(data,2)-1);
pw.println(line);
for index=1:length(data)
    disp(index);
    line=num2str(full(data(index,:)));
    pw.println(line);
end
pw.flush();
pw.close();

Here data is an extremely large sparse matrix.

Answer 4

Did you try partitioning it ?

I mean try calling full() on the 1000 first rows (or 5000) and then repeat the process if it works.

Answer 5

Use the find function to get the indices of non-zero elements...

idcs = find(data);
vals = data(idcs);
...save the index vector and value vector in whatever format you want...

If you want, you can use ind2sub to convert the linear indices to row, column subscripts.

If you need to recreate a sparse matrix in matlab from subscripts + values, use spconvert.

Answer 6

dlmwrite - Write matrix to ASCII-delimited file Syntax

dlmwrite(filename, M)

dlmwrite(filename, M, 'D')

dlmwrite(filename, M, 'D', R, C)

dlmwrite(filename, M, 'attrib1', value1, 'attrib2', value2, ...)

dlmwrite(filename, M, '-append')

dlmwrite(filename, M, '-append', attribute-value list)

Answer 7

If this is pretty much a one time deal, then I would just iterate through the matrix and write the matrix to an ASCII file by brute force, or else use @Veynom's suggestion and call full() on a subset of rows. It may take a while, but it will probably be done faster than it might take to learn how to read in a .mat file outside of the MATLAB environment.

If this is something you need to do on a recurring basis, then I would take @Vebjorn's advice and use a library to read the .mat file.

Answer 8

Use this script: msm_to_mm.m, writes an MATLAB sparse matrix to an MatrixMarket file.

And This thread may also be useful.