How to read EBCDIC data with a non standard codepage, and not mess up numbers?

Question

Here is one for the old(er) hands :-)

I\'m reading a binary dump from a mainframe DB2 table. The table has varchar, char, smallint, integer and float columns. To m

Answer 1

You can't read something like an EBCDIC file dump as a stream. The StreamReader class is a type of TextReader and exists for reading characters. You're reading a record -- a complex data structure containing mixed binary and text.

You need to do the reads with a FileStream and read blocks of octets as needed. You'll need some trivial helper methods like these:

private byte[] ReadOctets( Stream input , int size )
{
    if ( size < 0 ) throw new ArgumentOutOfRangeException() ;

    byte[] octets      = new byte[size] ;
    int    octets_read = input.Read( octets , 0 , size ) ;

    if ( octets_read != size ) throw new InvalidDataException() ;

    return octets ;
}

public string readCharVarying( Stream input )
{
    short    size        = readShort( input ) ;

    return readCharFixed( input , size ) ;
}

public string readCharFixed( Stream input , int size )
{
    Encoding e           = System.Text.Encoding.GetEncoding(20424) ;
    byte[]   octets      = ReadOctets( input , size ) ;
    string   value       = e.GetString( octets ) ;

    return value ;
}

private short readShort( Stream input )
{
    byte[] octets            = ReadOctets(input,2) ;
    short  bigEndianValue    = BitConverter.ToInt16(octets,0) ;
    short  littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

private int readInt( Stream input )
{
    byte[] octets            = ReadOctets(input,4) ;
    int    bigEndianValue    = BitConverter.ToInt32(octets,0) ;
    int    littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

private long readLong( Stream input )
{
    byte[] octets            = ReadOctets(input,8) ;
    long   bigEndianValue    = BitConverter.ToInt64(octets,0) ;
    long   littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

The IBM mainframe typically has fixed or variable length records in its file system. Fixed length is easy: you just need to know the record length and you can read all the bytes for the record in a single call to the Read() method, then convert the pieces as needed.

Variable length records are a little trickier, they start with 4-octet record descriptor word, consisting of 2-octet (16-bit) logical record length, followed by a 2-octet (16-bit) 0 value. the logical record length is exclusive of the 4-octet record descriptor word.

You might also see variable, spanned records. These are similar to variable length records, except that the 4-octet prefix is a segment descriptor word. the first 2 octets contains the segment length, the next octet identifies the segment type and the last octet is NUL (0x00). Segment types are as follows:

• 0x00 indicates a complete logical record
• 0x01 indicates that this is the first segment of a spanned record
• 0x10 indicates that this is the last segment of a spanned record
• 0x11 indicates that this is an "internal" segment of a spanned record, that is, a "Segment of a multisegment record other than the first or last segment."

You can treat variable length and variable spanned records as identical. To read one of these, you first need to parse out the segment/record/descriptor word and read/assemble the complete record into a byte[] from its constituent segment(s), then do whatever needs to be done to convert that byte[] into a form that you can use.