JDBC ResultSet internal mechanism of fetching large datasets

问题

Does JDBC result set fetch all data in one network call for a SQL query? Consider the query select * from table where timestamp > 1597937895. Now there are more than 1 million rows for this query. Does result set fetch all the rows in one network call? Or does it fetch batch of rows as and when the result set is read? Because I need to look at memory usage as well. Hence clarifying. I am known by the fact that ResultSet fetches all data in one network call. Is this is the only behaviour or is there any other way to tell result set to fetch data in batches?

回答1:

The exact behaviour for fetching rows and batching varies by database system and driver. Some will always batch, some will - by default - fetch all rows at once, and for some it depends on the result set type or other factors.

By default, the MySQL Connector/J driver will fetch all rows in memory on execute. This can be changed to either a row-streaming or a cursor-based fetch using a batch size, as documented on JDBC API Implementation Notes under Resultset:

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. If you are working with ResultSets that have a large number of rows or large values and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.

To enable this functionality, create a Statement instance in the following manner:

stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
              java.sql.ResultSet.CONCUR_READ_ONLY); 
stmt.setFetchSize(Integer.MIN_VALUE); 

The combination of a forward-only, read-only result set, with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this, any result sets created with the statement will be retrieved row-by-row.

[.. but do read the caveat ..]

Another alternative is to use cursor-based streaming to retrieve a set number of rows each time. This can be done by setting the connection property useCursorFetch to true, and then calling setFetchSize(int) with int being the desired number of rows to be fetched each time:

conn = DriverManager.getConnection("jdbc:mysql://localhost/?useCursorFetch=true", "user", "s3cr3t");
stmt = conn.createStatement();
stmt.setFetchSize(100);
rs = stmt.executeQuery("SELECT * FROM your_table_here");

来源：https://stackoverflow.com/questions/63981002/jdbc-resultset-internal-mechanism-of-fetching-large-datasets