Combine Multiple Record Types in SSIS Data Flow

问题

I can use Integration Services to extract and transform data that is, for example, contained in a single type of fixed format record in an import flat file. I can also handle multiple types of fixed format records in a single file, as long as I am able to send them to different destinations.

But what if I need to associate two (or more) records from an input file for each record that is going to a single destination? The only way I know that these records belong together is that they occur next to each other in the input file. The Data Flow will process one record after another. I can't use an SSIS package variable to count records, because the package variables are only updated after the Data Flow has finished.

Here is a sample input file:

HDR00120140501      
DTL001JOAN     0.00 
DTL002     30.00 ABC
DTL001DAVE    11.00 
DTL002     21.85 DEF
DTL001BERT    50.00 
DTL002      0.00 GHI
TRL001

I need to combine data from the DTL001 and DTL002 detail lines for a single output record. In addition, I need to include information from the HDR001 header line (in this case a date value). So the destination table looks like:

CREATE TABLE TestImport (
  ID int IDENTITY(1,1) NOT NULL,
  ImportDate datetime NULL,
  Name char(4) NULL,
  Amount1 decimal(18, 2) NULL,
  Amount2 decimal(18, 2) NULL,
  Desc char(3) NULL )

The six input detail records, plus the header, will yield three records in the destination table:

ImportDate Name Amount1 Amount2 Desc
---------- ---- ------- ------- -----
2014-05-01 JOAN 0.00    30.00   ABC
2014-05-01 DAVE 11.00   21.85   DEF
2014-05-01 BERT 50.00   0.00    GHI

This is a solved problem that is posted to assist anyone who may encounter a similar problem.

回答1:

The detail records can be combined using a combination of Script, Conditional Split, and Merge Join components.

The Script component adds a Detail Record Count that is incremented for each DTL001 record that it encounters. This provides a Detail Record Count column that can be used for sorting and joining the DTL001 and DTL002 records that are next to each other in the input file.

The Flat File Connection Manager and Source are used to break each input record into RecordType, RecordSubType, and RecordData columns. For instance, the first three records are broken up into the following pieces:

"HDR", "001", "20140501      "
"DTL", "001", "JOAN     0.00 "
"DTL", "002", "     30.00 ABC"

Next, the Derived Header Columns component adds HeaderRecordCount, DetailRecordCount, and ImportDate columns. These are initially unpopulated, but they provide a place for the Script component to store the Header and Detail counts, as well as any information from the header that will be needed for each row of the destination table. This Script could also handle files with multiple sections, where each section has a new HDR001 header record:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    static int _HeaderRecordCount = 0;
    static int _DetailRecordCount = 0;
    static DateTime _ImportDate = DateTime.MinValue;
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        if (Row.RecordType == "HDR" && Row.RecordSubType == "001")
        {
            _HeaderRecordCount = _HeaderRecordCount + 1;
            _ImportDate = DateTime.ParseExact(Row.RecordData.Substring(0, 8), "yyyyMMdd", System.Globalization.CultureInfo.InvariantCulture);
        }
        else if (Row.RecordType == "DTL" && Row.RecordSubType == "001")
        {
            _DetailRecordCount = _DetailRecordCount + 1;
        }
        Row.HeaderRecordCount = _HeaderRecordCount;
        Row.DetailRecordCount = _DetailRecordCount;
        Row.ImportDate = _ImportDate;
    }
}

The Conditional Split component then sends the DTL001 and DTL002 records down separate paths. The split conditions are:

RecordType == "DTL" && RecordSubType == "001"
RecordType == "DTL" && RecordSubType == "002"

These paths are each Sorted on the DetailRecordCount column, and the Merge Join combines them using that column. All of the records from the 001 path are passed through to the Merge Join output, and the RecordData column from the 002 path is also passed to a separate output column (e.g., RecordData002).

Now the contents of both Flat File records are available in a single Data Flow record. The Derived Detail Columns component is used to pull out any required information, and the resulting columns are sent to the destination table.

来源：https://stackoverflow.com/questions/23942421/combine-multiple-record-types-in-ssis-data-flow

标签

ssis

sql-server-2012