Sort a file with huge volume of data given memory constraint

前端 未结 12 1040
暖寄归人
暖寄归人 2020-11-28 21:47

Points:

  • We process thousands of flat files in a day, concurrently.
  • Memory constraint is a major issue.
  • We use thread for each file process
12条回答
  •  栀梦
    栀梦 (楼主)
    2020-11-28 22:44

    If you can move forward/backward in a file (seek), and rewrite parts of the file, then you should use bubble sort.

    You will have to scan lines in the file, and only have to have 2 rows in memory at the moment, and then swap them if they are not in the right order. Repeat the process until there are no files to swap.

提交回复
热议问题