ff package write error

不羁岁月 提交于 2019-12-01 23:22:14

This error message indicates that you have too many open files. In ff, every column in your ffdf is a file. You can only have a limited number of files open - and you have hit that number. See my reply on Any ideas on how to debug this FF error?.

So in your case, using simply read.table.ffdf won't work because you have 139352 columns. It is possible however to import it in ff but you need to be carefull when opening columns while getting data in RAM to avoid this issue.

N8TRO

Your data set really isn't that big.. It might help if you said something about what you're trying to do with it. this might help: Increasing Available memory in R or if that doesn't work, the data.table package is VERY fast and doesn't hog memory when manipulating data.tables with the := operator.
and as far as read.table.ffdf, check this out.. read.table.ffdf tutorial, if you read carefully, it gives hints and details about optimizing your memory usage with commands like gc() and more.

I recently encountered this problem with a data frame that had ~ 3,000 columns. The easiest way to get around this is to adjust the maximum number of files allowed open for your user account. The typical system is set to ~ 1024 and that is a very conservative limit. Do note that it is set to prevent resource exhaustion on the server.

On Linux:

Add the following to your /etc/security/limits.conf file.

youruserid hard nofile 200000 # you may enter whatever number you wish here youruserid soft nofile 200000 # whatever you want the default to be for each shell or process you have running

On OS X:

Add or edit the following in your /etc/sysctl.con file. kern.maxfilesperproc=200000 kern.maxfiles=200000

You'll need to log out and log back in but then the original poster would be able to use the ffdf to open his 139352 column data frame.

I've posted more about my run-in with this limit here.

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!