问题
I'm trying to test what the memory limitations in the current R version is.
runtest <- function(size) {
x <- "testme"
while(0<1) {
x <- c(x, x)
size <<- object.size(x) # size of x when fail
}
}
By running runtest(size) in the console on my laptop, I get the following error:
> runtest(size)
Error: cannot allocate vector of size 4.0 Gb
In addition: Warning messages:
1: In structure(.Call(C_objectSize, x), class = "object_size") :
Reached total allocation of 7915Mb: see help(memory.size)
2: In structure(.Call(C_objectSize, x), class = "object_size") :
Reached total allocation of 7915Mb: see help(memory.size)
3: In structure(.Call(C_objectSize, x), class = "object_size") :
Reached total allocation of 7915Mb: see help(memory.size)
4: In structure(.Call(C_objectSize, x), class = "object_size") :
Reached total allocation of 7915Mb: see help(memory.size)
> size
2147483736 bytes
>
This size looks very close to the 2^31-1 limit that people have mentioned before. So then I tried running the same code on our upgraded desktop with 128GB of RAM and set the variable in the shortcut for the 64 bit version to the max memory usage of 100GB. This is the new error I get:
Error in structure(.Call(C_objectSize, ), class = "object_size"):
long vectors not supported yet: unique.c: 1720
> size
8589934680 bytes
>
Does this 8.5GB limit have anything to do with running in Windows O/S (specifically Windows 7 Enterprise edition)? I think the R help file (http://stat.ethz.ch/R-manual/R-devel/library/base/html/Memory-limits.html) explains this, but I'm having trouble understanding what it's saying (not my area of expertise).
回答1:
Looking at the source of size.c and unique.c it looks like the hashing used to improve object.size doesn't support long vectors yet:
/* Use hashing to improve object.size. Here we want equal CHARSXPs,
not equal contents. */
and
/* Currently the hash table is implemented as a (signed) integer
array. So there are two 31-bit restrictions, the length of the
array and the values. The values are initially NIL (-1). O-based
indices are inserted by isDuplicated, and invalidated by setting
to NA_INTEGER.
*/
Therefore, it is object.size that is choking. How about calling numeric(2^36) to see if you can create a such a large object, (should be 64GB).
回答2:
The maximum vector length in recent versions of R is 2^53-1 because there are 53 bits in the mantissa of the double storage mode. The maximum extent of each dimension of a matrix or array is still 2^32-1 because dimension values are still based on the integer storage mode. I thought I might get more information from news() but didn't get as much as I thought I would. There was quite a bit of discussion about this on the r-devel mailing list and I would use MarkMail's archive search if I needed more.
?double
?integer
db <- news()
str(db)
db$Text[grepl("vector", db$Text) & grepl("length", db$Text)]
# only slsightly informative
db$Text[grepl("vector", db$Text) & grepl("long", db$Text)][3]
[1] "There is support for vectors longer than 2^31 - 1 elements. This\napplies to raw, logical, integer, double, complex and character\nvectors, as well as lists. (Elements of character vectors remain\nlimited to 2^31 - 1 bytes.)"
来源：https://stackoverflow.com/questions/21140818/long-vector-not-supported-yet-error-in-r-windows-64bit-version