Tricks to manage the available memory in an R session

前端 未结 27 2154
情深已故
情深已故 2020-11-22 01:23

What tricks do people use to manage the available memory of an interactive R session? I use the functions below [based on postings by Petr Pikal and David Hinds to the r-he

27条回答
  •  面向向阳花
    2020-11-22 01:52

    1. I'm fortunate and my large data sets are saved by the instrument in "chunks" (subsets) of roughly 100 MB (32bit binary). Thus I can do pre-processing steps (deleting uninformative parts, downsampling) sequentially before fusing the data set.

    2. Calling gc () "by hand" can help if the size of the data get close to available memory.

    3. Sometimes a different algorithm needs much less memory.
      Sometimes there's a trade off between vectorization and memory use.
      compare: split & lapply vs. a for loop.

    4. For the sake of fast & easy data analysis, I often work first with a small random subset (sample ()) of the data. Once the data analysis script/.Rnw is finished data analysis code and the complete data go to the calculation server for over night / over weekend / ... calculation.

提交回复
热议问题