Retrieving large number of rows (more than 10 mil) in asp.net mvc application

问题

I am working on asp.net mvc application and it provides the functionality of reading from ORACLE database using DATAREADER and present those rows to the user (sometimes up to 10 mil). The datareader read operation throws out of memory exception after reading about 900,000 rows.

I was discussing this issue with my colleague and he suggested that I should use connectionless paradigm (may be Entity framework) or stored procedure and bring data in chunks.

I wonder if there is someone out there who can authoritatively say which is the best way to accomplish above issue.

回答1:

Don’t retrieve all the rows to memory and perform the paging

•Not all the users visits 2nd page

•So your data in memory will be unused

If you are having more records use SQL side paging, you can use Row_number() function to perform paging in SQL side.

You can also use ORM frameworks to access the data, they always provides best approaches to perform data related operations.

I prefer to use Peta Poco, it has a method to retrieve page wise data.

var result=db.Page<article>(1, 20, // <-- page number and items per page
"SELECT * FROM articles WHERE category=@0 ORDER BY date_posted DESC", "coolstuff");

http://www.toptensoftware.com/petapoco/

来源：https://stackoverflow.com/questions/18214873/retrieving-large-number-of-rows-more-than-10-mil-in-asp-net-mvc-application