Edit: huge thanks to the users below for great contributions and to Gregor for benchmarking.
Say I have a matrix filled with integer values like this...
Perhaps you may be able to use a distance function here using the row and column indices of the matrix elements.
# data
(mat <- matrix(16:31, 4, 4))
[,1] [,2] [,3] [,4]
[1,] 16 20 24 28
[2,] 17 21 25 29
[3,] 18 22 26 30
[4,] 19 23 27 31
# find distances between row and column indexes
# interested in values where the distance is one
w <- which(mat==mat, arr.ind=TRUE)
d <- as.matrix(dist(w, "maximum", diag=TRUE, upper=TRUE))
# extract neighbouring values for each element
# extract where max distance is one
a <- apply(d, 1, function(i) mat[i == 1] )
names(a) <- mat
a
$`16`
[1] "17" "20" "21"
$`17`
[1] "16" "18" "20" "21" "22"
$`18`
[1] "17" "19" "21" "22" "23
... ....
... ....
Needs tidied, but maybe gives an idea