Converting data stored in Fortran 90 binaries to human readable format

Question

In your experience, in Fortran 90, what is the best way to store large arrays in output files? Previously, I had been trying to write large arrays to ASCII text files. For example, I would do something like this (thanks to the recommendation at the bottom of the page In Fortran 90, what is a good way to write an array to a text file, row-wise?):

PROGRAM testing1
  IMPLICIT NONE
  INTEGER :: i, j, k
  INTEGER, DIMENSION(4,10) :: a

  k=1
  DO i=1,4
    DO j=1,10
      a(i,j)=k
      k=k+1
    END DO
  END DO

  OPEN(UNIT=12, FILE="output.txt", ACTION="WRITE", STATUS="REPLACE")
  DO i=1,4
    DO j=1,10
      WRITE(12, "(i2,x)", ADVANCE="NO") a(i,j)
    END DO
    WRITE(12, *)
  END DO
  CLOSE(UNIT=12)
END PROGRAM testing1

This works, but as pointed out by the topmost reply at In Fortran 90, what is a good way to write an array to a text file, row-wise?, writing large arrays to text files is very slow and creates files that are somewhat larger in size than is necessary. The poster there recommended instead writing to an unformatted Fortran binary, using something like:

PROGRAM testing2
  IMPLICIT NONE
  INTEGER :: i, j, k
  INTEGER, DIMENSION(4,10) :: a

  k=1
  DO i=1,4
    DO j=1,10
      a(i,j)=k
      k=k+1
    END DO
  END DO

  OPEN(UNIT=13, FILE="output.dat", ACTION="WRITE", STATUS="REPLACE", &
      FORM="UNFORMATTED")
  WRITE(13) a
  CLOSE(UNIT=13)
END PROGRAM testing2

This seems to work, and is indeed much faster and results in smaller file sizes, as promised by the reply here. However, what do I do if I would like to be able to later work with the data stored in Fortran binary (e.g., output.dat above) and analyze its contents? For example, what if I want to open the array stored in the binary in a program such as Microsoft Excel?

When I mentioned matlab in my previous post, the reply suggested that I open the binary as a hexadecimal file and figure out and extract the records from there. But, I am nervous that I am getting into deep water since I have no prior experience in hexadecimal sleuthing. When I asked on the matlab board (here: http://www.mathworks.com/matlabcentral/answers/12639-advice-on-reading-an-unformatted-fortran-binary-file-into-matlab) about reading Fortran files into matlab, the person there suggested that using Fortran stream might be easy. But is Fortran stream (i.e., using the directive ACCESS="STREAM" in the OPEN command) likely to be similar in time and file size to the ASCII text file that I created in my first example above?

Or, do you know if there is any other software that can automatically read Fortran binaries into some sort of human readable form? (Or, do you know of any good tutorials on either hexadecimal sleuthing or Fortran stream?)

Thank you very much for your time.

Answer 1

Stream is a choice independent of the choice of formatted / unformatted -- one is "access", the other "format" The default for Fortran I/O is record oriented access. The typical approach of a Fortran compiler for records (at least unformatted) to write a 4-byte record length before and after each record. (The "after" is to make reading backwards easier.) Using a hex edit you could verify these extra data items that I described and skip them in MatLab. But they are not part of the language standard and are not portable and are certainly not obvious in other languages. If you select stream and unformatted you will just get the raw sequence of bytes corresponding to your data items -- no extra data items to worry about in the other language! In my experience this output tends to be fairly easy to read in other languages (not tried in MatLab). If this is a small & simple project with portability of the files to other computers not an issue, I would probably use this approach (stream & unformatted) rather than a file format specification such as HDF5 or FITS. I'd write the array as write (13) a, as in your final example. Depending on the other language, you might have to transpose the dimensions. If this is a major and long-lived project with portability a concern, then a portable and standard file interface is worth considering.

I don't know whether any of these formats can be read from Excel. More research.... You might have to write a program to read the binary file of whatever format and output a file in a format that Excel understands.

Answer 2

(converting comment into an answer for posterity)

Are you specifically trying to get information into Matlab? If you are, I highly recommend HDF5. This is the portable binary format you have been looking for.

For converting a Fortran binary to HDF5, you're going to have to read in the original Fortran binary and then write out the same data to an HDF5 file. If you have the Fortran source, this should be pretty easy. Allocate your arrays, make sure you read the arrays in the same order as you wrote them and then write out your new shiny HDF5 file.

The HDF5 group has tutorials with examples in C and Fortran. There is likely an example very close to what you're trying to do. When you build HDF5, make sure to manually enable Fortran support. It is disabled by default.

Answer 3

 %In MATLAB

 fid=fopen('YOUR_FILE.direct','r');  %Fortran Direct ACCESS
 frewind(fid);

 tbb=ones(367,45203);

     for i =1:367
          temp=fread(fid,[45203],'single');
          tbb(i,:)=temp; 
     end

fclose(fid)