My application is a stand-alone Swing client invoking EJB Stateless Session beans thanks to classical JNDI lookup and RMI-IIOP method calls. It is started as a Java WebStart
To sum up the context: our deployment is in production since years with IBM WebSphere running on Linux and application deployed thanks to Java WebStart running on Sun JavaSE 6 with IBM ORB included and configured to connect without any authentication. Now we want to enable Kerberos authentication and single-sign-on over RMI-IIOP, supported since WebSphere 6 (I think).
Here are now pieces of answer.
Since WebSphere 7, a new concept has been introduced to configure security aspects on a per server basis: security domain. Theoretically any option that has not been changed in a security domain inherits from the global security section.
When testing Kerberos setup, we have created a dedicated security domain for our test server, to avoid troubles with other servers running in the cell.
BUT even if Kerberos is enabled in global security, it is not inherited/enabled for a server configured with its own security domain.
As soon as we run our test server with the default global security where Kerberos options are visible and enabled, then Kerberos authentication has began to work with IBM JavaSE 6 executed from a cmd bat script with usual ClassPath and all properties declared in documentation.
To note: the JNDI Context.SECURITY_AUTHENTICATION option is never set. After decompilation, the only available values for IBM ORB are none, simple and strong but strong has no implementation yet.
Another point: according to generated log, IBM ORB is not able to work with file:/C:/temp/sas.client.config as value for com.ibm.CORBA.ConfigURL. It MUST be an URI and not a file path. We even got DNS lookup failure to resolve C hostname ! Arff. All documentation examples are Unix based with file:/path/to/sas.client.config so we made many trials before delivering that file from an HTTP server.
Now the Java WebStart part of the deployment:
the same original JNLP without any security and no Kerberos settings works perfectly with both Oracle JavaSE 6 and IBM Java 6
with WebSphere security enabled and Kerberos in JNLP (and only that change set), IBM ORB running on IBM Java 6 complains with NoClassDefFoundError about ffdc log manager implementation that is (still/always) available in ClassPath. It really sounds like a code incompatibility with Java WebStart secured ClassLoader hierarchy.
with Kerberos JNLP, IBM ORB running on Oracle JavaSE 6 seems to simply ignore security settings and connects anonymously as usual.
So a first step is now working: IBM Java 6 started from command-line but investigations are not over to reach our goal: Kerberos with Oracle JavaSE 6 in Java WebStart context.