Tomcat startup fails due to 'java.net.SocketException Invalid argument' on Mac OS X

Question

We have an application that runs on Tomcat 6 (6.0.35.0 to be precise), and most of our engineers on Mac OS are having problems starting Tomcat due to the socketAccept call i

Answer 1

Have you tried turning on JNI debugging with -Xcheck:jni? Interestingly the Oracle documentation uses a PlainSocketImpl.socketAccept error as an example of when to use this.

Note also that the implication of Bug 7131399 is that the JNI uses poll() on most platforms but select() on Mac OS due to a problem with poll() on the Mac. So maybe select() is broken too. Digging in a bit further, select() will return EINVAL if "ndfs is greater than FD_SETSIZE and _DARWIN_UNLIMITED_SELECT is not defined." FD_SETSIZE is 1024 and it sounds like you have a ton of applications loading up, so perhaps it all filters down to waiting on more that 1024 FDs at one time.

For extra credit, see if the related (supposedly fixed) Java bug is in fact fixed on your machine. The bug report has pointers to test cases.

---

Thanks to Old Pro's answer, I confirmed that the select() FD_SETSIZE limitation is the cause. I located an existing bug for this limitation:

https://bugs.openjdk.java.net/browse/JDK-8021820

The problem can be reproduced with the following code:

import java.io.*;
import java.net.*;

public class SelectTest {
  public static void main(String[] args) throws Exception {
    // Use 1024 file descriptors. There'll already be some in use, obviously, but this guarantees the problem will occur
    for(int i = 0; i < 1024; i++) {
      new FileInputStream("/dev/null");
    }
    ServerSocket socket = new ServerSocket(8080);
    socket.accept();
  }
}

Almost a year later, Java 7u60 has a fix this problem:

http://www.oracle.com/technetwork/java/javase/2col/7u60-bugfixes-2202029.html

I also discovered the Tomcat's WebappClassLoader closes file handles after 90 seconds, which explains why setting break points prevented the issue from occurring.