Fragmented XML Bulk Load to SQL Server in C#

问题

I have an XML that contain information resulted from scanning systems on different domains. The XML corresponds to the tables in database that are nested as follows:

Domains
   Computers
      Volumes
          Folders
              Files

My goal is to load the XML into the corresponding tables. Since one single XML file would be so large to load into database, I have to chunk it into smaller one. How can I format the XMLs so the uploader knows one file is a continue of the last file and it does not generate additional keys for a parent node that is already added. Does any body any experience doing this? Is this the fastest way to upload? Many Thanks! Here is the code that I use to upload the xml:

        static void BulkLoadXML()
    {
        try
        {
            string sPath = @"C:\XMLFiles\";


            string FileName, xsdPath, xmlPath;
            FileName = "TestAuto";

            xmlPath = sPath + FileName + ".xml";
            xsdPath = sPath + FileName + ".xsd";

            SQLXMLBULKLOADLib.SQLXMLBulkLoad4 objBL = new SQLXMLBULKLOADLib.SQLXMLBulkLoad4();

            objBL.ConnectionString = @"Provider=sqloledb; server=srv1;database=MyTest;User ID=sa;Password=psw;Connection Timeout=60";

            objBL.ErrorLogFile = "error.xml";
            objBL.KeepIdentity = false;

            objBL.Execute(xsdPath, xmlPath);
        }
        catch (Exception e)
        {
            Console.WriteLine(e.ToString());
            Console.Read();
        }
        Console.Read();
    }

回答1:

Perhaps a bulk insert is in order here? I think in C# you don't even need to write to a text file first. Use an XMLReader to get the XML into memory and then bulk insert it. Read the following blog post for info on bulk insert in C#: http://blogs.msdn.com/b/nikhilsi/archive/2008/06/11/bulk-insert-into-sql-from-c-app.aspx

回答2:

I did a a lot of research and here is an idea that partially solves the problem:

http://rakeshbajania.wordpress.com/2011/01/01/prevent-duplicate-entry-when-using-sqlbulkcopy/

So the idea is to define a unique index for tables and set the IGNORE_DUP_KEY to ON. The only problem is that the DB returns the error and does not upload the children nodes. In the end, I decided to create temp tables that are loaded by xml chunks one by one and empty them after sending their records to the main tables using stored procedures.

来源：https://stackoverflow.com/questions/24679818/fragmented-xml-bulk-load-to-sql-server-in-c-sharp

标签

sql

sql-server

xml

bulk-load