Error in heatmap.2 (gplots)

Question

Ive moved on to a new server and Installed R version 3.0 on it. (gplots library was no longer available for 2.14)

Using a script that worked for version 2.14 I now encounter a problem generating a heatmap.

In R version 3 I get an error:

Error in lapply(args, is.character) : node stack overflow
Error in dev.flush() : node stack overflow
Error in par(op) : node stack overflow

In R version 2.14 I get an error:

Error: evaluation nested too deeply: infinite recursion / options(expressions=)?

Which I can resolve by increasing the options(expressions=500000)

In R version 3 increasing this option does not resolve the issue. And Im still stuck with the same error.

The script is the same for both:

y=read.table("test", row.names=1, sep="\t", header=TRUE)
hr <- hclust(dist(as.matrix(y)))
hc <- hclust(dist(as.matrix(t(y))))
mycl <- cutree(hr, k=7); mycolhc <- rainbow(length(unique(mycl)), start=0.1, end=0.9); mycolhc     <- mycolhc[as.vector(mycl)] 

install.packages("gplots")
library("gplots", character.only=TRUE)
myheatcol <- redgreen(75)

pdf("heatmap.pdf")
heatmap.2(as.matrix(y), Rowv=as.dendrogram(hr), Colv=as.dendrogram(hc), col=myheatcol,scale="none", density.info="none", trace="none", RowSideColors=mycolhc, labRow=FALSE)
dev.off()

Where "test" is a tdl file with headers and row names and a 40*5000 0/1 matrix

Any help would be appreciated

PS: When I reduce my data set to 2000 lines I no longer get the error.

PSS: Increasing the dataset to 2500 lines resulted in the same error; However, removing all non-informative lines (all 1s) left me with 3700 lines. Using this data set did not result in the error.

Answer 1

I'm the author of the gplots package. The 'node stack overflow' error occurs when a byte-compiled function has too many recursive calls.

In this case, it occurs because the function that plots dendrogram objects (stats:::plotNode) is implemented using a recursive algorithm and the dendrogram object is deeply nested.

Ultimately, the correct solution is to modify plotNode to use an iterative algorithm, which will prevent the recursion depth error from occuring.

In the short term, it is possible to force stats:::plotNode to be run as interpreted code rather then byte-compiled code via a nasty hack.

Here's the recipe:

## Convert a byte-compiled function to an interpreted-code function 
unByteCode <- function(fun)
    {
        FUN <- eval(parse(text=deparse(fun)))
        environment(FUN) <- environment(fun)
        FUN
    }

## Replace function definition inside of a locked environment **HACK** 
assignEdgewise <- function(name, env, value)
    {
        unlockBinding(name, env=env)
        assign( name, envir=env, value=value)
        lockBinding(name, env=env)
        invisible(value)
    }

## Replace byte-compiled function in a locked environment with an interpreted-code
## function
unByteCodeAssign <- function(fun)
    {
        name <- gsub('^.*::+','', deparse(substitute(fun)))
        FUN <- unByteCode(fun)
        retval <- assignEdgewise(name=name,
                                 env=environment(FUN),
                                 value=FUN
                                 )
        invisible(retval)
    }

## Use the above functions to convert stats:::plotNode to interpreted-code:
unByteCodeAssign(stats:::plotNode)

## Now raise the interpreted code recursion limit (you may need to adjust this,
##  decreasing if it uses to much memory, increasing if you get a recursion depth error ).
options(expressions=5e4)

## heatmap.2 should now work properly 
heatmap.2( ... )

Answer 2

In another post this is from stats:::midcache.dendrogram's function setmid. setmid calls itself recursively, and this recursion might be too deep -- probably the dendrogram is too dense to make any sense visually? You see where the error occurs by looking at the last few lines of traceback() after the error occurs.

To make further progress with this, you need to be able to provide a minimal reproducible example (using heatmap rather than heatmap.2, or even more refined based on your interpretation of traceback()) , perhaps by making the data file available, or by providing a recipe to simulate the data (m <- matrix(runif(1000), 40) ?) in a way that reliably reproduces the error.

Answer 3

This problem ("node stack overflow" error while using heatmap.2 function) occurs due to too many identical values in a specific column in your matrix, which causes recursion issue on R producing the error.

What I can suggest (which at least how I solved my very exact problem for my data) is to produce random numbers around the identical numbers and replace them with the original numbers in the matrix:

for (i in 1:nrow(my_matrix)) {
   if (my_matrix[i,my_column]=="100") { # assume i have too many 100 in my_column
      my_matrix[i,my_column]=runif(1,99,101) # replace 100 with nearby values randomly
    }
}

In this way, the heatmap is created without any issue since there are not too many identical numbers anymore, and also it virtually doesn't affect your matrix since you can choose a very small interval for random number generation around your identical value which will still reflect the original values with invisible color changes on the heatmap.