Can a C# program read a text file into memory and then pass that object to a method that requires a filename?

Question

In a C# program I am importing a large text file (300mb) into a MySQL database via the MySqlBulkLoader function of the MySql .Net Connector.

The import takes quite a long time and results in almost 100% disk usage on the Windows 2003 Server it is running on. In an attempt to speed up the import the program nows splits the big file into smaller chunks.

Would it be possible to read the small file chunk (8mb) into memory (ie array) and then pass it to the MySQLBulkLoader as a file?

The bulk loader looks for a filename path:

 MySql.Data.MySqlClient.MySqlBulkLoader myBulk = new MySql.Data.MySqlClient.MySqlBulkLoader(connection);                
 myBulk.Timeout = 10 * 60; /
 myBulk.TableName = "some_table";                
 myBulk.Local = true;
 myBulk.LineTerminator = @"\n";
 myBulk.FileName = aFile.FullName;
 myBulk.FieldTerminator = "";

Answer 1

Memory isn't a file, so the short answer is no. The alternatives are:

1. Read through the file, writing it out as a temporary file (System.IO.Path.GetTempFileName() is your friend here, for the name to give the partial file) and passing that filename to MySqlBulkLoader
2. Use a "RAM Disk" tool to create a memory based disk to put a copy of the full 300Mb file on, then pass that files path to MySqlBulkLoader.

Answer 2

The following class accepts that it is not possible and writes the DataTable to disk before bulkloading.

It may not be suited to all circumstances, but it suited my needs at the time.

using MySql.Data.MySqlClient;
using System.Data;
using System.IO;
using System.Text;

namespace ImportDatabase
{
    class DataTableToMySql
    {
        public MySqlConnection Connection { get; set; } 
        public DataTable SourceDataTable { get; set; } 
        public string FieldTerminator { get; set; }
        public string LineTerminator { get; set; }

        public DataTableToMySql(MySqlConnection conn, DataTable table)
        {
            FieldTerminator = "\t";
            LineTerminator = "\n";

            Connection = conn;
            SourceDataTable = table;
        }

        public void Execute()
        {
            string fileName = Path.GetTempFileName();

            try
            {
                byte[] fieldTerm = Encoding.UTF8.GetBytes(FieldTerminator);

                byte[] lineTerm = Encoding.UTF8.GetBytes(LineTerminator);

                PrepareFile(fileName, fieldTerm, lineTerm);

                LoadData(fileName);
            }
            finally
            {
                File.Delete(fileName);
            }
        }

        private void LoadData(string fileName)
        {
            MySqlBulkLoader bl = new MySqlBulkLoader(Connection);

            bl.FieldTerminator = FieldTerminator;

            bl.LineTerminator = LineTerminator;

            bl.TableName = SourceDataTable.TableName;

            bl.FileName = fileName;

            bl.Load();
        }

        private void PrepareFile(string fileName, byte[] fieldTerm, byte[] lineTerm)
        {
            using (FileStream fs = new FileStream(fileName, FileMode.Append))
            {
                foreach (DataRow row in SourceDataTable.Rows)
                { 
                    int i = 0;

                    foreach (object val in row.ItemArray)
                    {
                        byte[] bytes;

                        if (val is DateTime)
                        {
                            DateTime theDate = (DateTime)val;
                            string dateStr = theDate.ToString("yyyy-MM-dd HH:mm:ss"); 
                            bytes = Encoding.UTF8.GetBytes(dateStr);
                        }
                        else
                            bytes = Encoding.UTF8.GetBytes(val.ToString());

                        fs.Write(bytes, 0, bytes.Length);

                        i++;

                        if (i < row.ItemArray.Length)
                            fs.Write(fieldTerm, 0, fieldTerm.Length);
                    }

                    fs.Write(lineTerm, 0, lineTerm.Length);
                }
            }
        }
    }
}