Why is the JDK NIO using so many anon_inode file descriptors?

Question

I'm using Sun's JDK 1.6.0_26 and NIO (with Netty) and in lsof I see hundreds of file descriptors that are anon_inode:

$ lsof -np 11225 | fgrep -w anon_inode
java    11225 nobody   57u     0000                0,9         0     1386 anon_inode
java    11225 nobody   61u     0000                0,9         0     1386 anon_inode
java    11225 nobody   65u     0000                0,9         0     1386 anon_inode
java    11225 nobody   69u     0000                0,9         0     1386 anon_inode
java    11225 nobody   73u     0000                0,9         0     1386 anon_inode
java    11225 nobody   77u     0000                0,9         0     1386 anon_inode
java    11225 nobody   81u     0000                0,9         0     1386 anon_inode
java    11225 nobody   86u     0000                0,9         0     1386 anon_inode
java    11225 nobody   89u     0000                0,9         0     1386 anon_inode
java    11225 nobody   93u     0000                0,9         0     1386 anon_inode
java    11225 nobody   97u     0000                0,9         0     1386 anon_inode
[...]

I couldn't find a clear explanation as to what an anonymous inode is, I looked at fs/anon_inodes.c in the Linux kernel's source tree and it seems that maybe epoll uses it, but I'm not sure why I would have so many. I do have multiple "epoll loops" and timer threads, but not nearly as many as my number of anon_inode.

Answer 1

It is indeed most probably epoll. Selectors use epoll by default starting with one of the early JDK 1.6.x releases and this selector impl uses more file descriptors than the old one in addition to the descriptors used by sockets themselves. You will also use more file descriptors if you register several Selectors with a single SocketChannel, for example separate selectors for reads and writes. If Netty uses NIO, it almost certainly makes use of selectors. In one app my file descriptors to sockets ratio was about 4 to 1 due to using epoll-based Selectors. It is possible to revert to the old selector implementation by changing the java.nio.channels.spi.SelectorProvider property and it will reduce the number of descriptors, but probably at the cost of performance (YMMV).

Answer 2

Perhaps these inodes are related to memory mapped "files" which would translate into Java's Direct ByteBuffers. The system call mmap(2) usually operates on files (using filehandles). But it also supports an option MAP_ANONYMOUS to manipulate the memory mapping without having an actual filehandle. This sounds like something which might need an "anonymous inode" internally.