Is a “Load DATA” without a file (i.e., in memory) possible for MySQL and Java?

拟墨画扇 提交于 2019-11-29 10:08:59

问题


I'm in the process of optimizing an import of ~10TB of Data into a MySQL database. Currently, I can import 2.9GB (+0.8GB index) in about 14 minutes on a current laptop. The process includes reading a data file (Oracle ".dat" export), parsing the data, writing the data into a CSV file and executing the "LOAD DATA LOCAL" sql command on it.

Is it possible to increase the import speed (without hardware changes)? Is there a way to remove the step of writing a file to the file system and letting MySQL read it again. Is it possible to stream the data in memory directly to MySQL (e.g., via the JDBC driver)?

Many thanks in advance, Joerg.


回答1:


It seems that from MySQL Connector/J JDBC driver version 5.1.3 onwards, you can hook up an InputStream reference, using com.mysql.jdbc.Statement.setLocalInfileInputStream() method, internally within your Java code, to 'pipe' your in-memory formatted string/text to the 'LOAD DATA INFILE' call. This means you do not have to write out and re-read a temporary file back from memory. Please refer to:

http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-implementation-notes.html (bottom of page)

The process is also outlined in this post:

http://jeffrick.com/2010/03/23/bulk-insert-into-a-mysql-database

O'reilly produced a PDF covering MySQL/JDBC performance gems, which refers to this.

There is also mention of it's usage with Hadoop (advanced Java topic).

Hope this all helps.

Cheers

Rich




回答2:


Actual working code for this was hard to come by, so here's some:

@Test
public void bulkInsert() throws SQLException {
    try(com.mysql.jdbc.Connection conn = (com.mysql.jdbc.Connection) dao.getDataSource().getConnection()) {

        conn.setAllowLoadLocalInfile(true);

        try(com.mysql.jdbc.Statement stmt = (com.mysql.jdbc.Statement) conn.createStatement()) {

            stmt.execute("create temporary table BasicDbTest_1 (phone integer)");

            String data = "8675309\n";
            stmt.setLocalInfileInputStream(new ByteArrayInputStream(data.getBytes()));

            stmt.execute("load data local infile '' into table BasicDbTest_1");

            try(ResultSet rs = stmt.executeQuery("select phone from BasicDbTest_1")) {
                Assert.assertTrue(rs.next());
                Assert.assertEquals(rs.getInt(1), 8675309);                 
            }
        }
    }
}


来源:https://stackoverflow.com/questions/3627537/is-a-load-data-without-a-file-i-e-in-memory-possible-for-mysql-and-java

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!