I need to build an installer that does the following:
You may also want to take a look at BitRock InstallBuilder, it provides built-in functionality to perform what you want. We will be happy to build a sample installer for you, just drop us a line.
I recommend downloading the source code for the build scripts of popular Java programs like limewire and follow their example. The end.
...Or you can do like I did:
All this was a fair chunk of work. I first did the Windows stuff then a couple years later did the Mac stuff.
You can integrate all of this in NSIS build scripts and Ant. I can't remember why I didn't use install4j. Probably I was too cheap :)
Commercial tools like InstallAnywhere can do that. After installing the JAR (and links and such), installers can check the registry if Java is already installed and proceed or not. As long as the installer supports all needed plattforms, you are fine.
If a commercial tool is ok, I can recommend install4j. 1, 2 and 4 are easy to do with it, but about 3 I'm not sure. Edit: You could probably achieve that by installing your app as a "service", which install4j supports for all platforms. Quoting from its Features page:
install4j offers full support for generating and installing services (daemons). For services, install4j generates service executables on Windows, startup items on Mac OS X and start/stop scripts on Unix/Linux platforms.
Edit: Regarding #2, JRE detection: install4 has pretty nice, flexible options for JRE bundling/detecting. You can include a JRE (statically or dynamically, shared or not) or prefer to use one already present, with the possibility to download a JRE from within your installer as a fallback. Check these docs for more: How Installers Find a JRE and JRE Bundles.
(More about my experiences with install4j.)
For other tools, you could check some of the earlier Java installer questions on SO:
You could use launch4j , it does not automatically install java however. But maybe there's a way to make that work by packaging the JRE for example.
Haven't heard of anything that works across all those platforms, but JSmooth might help (fits your needs on Windows). I've used it about 2 years ago and it worked fine.