As others have mentioned is is definitely possible to create good looking cross platform GUI in Java on both Windows and Mac. However if you want your app to blend in and behave in a way that makes it "feel" like an application designed from the ground up for the platform it is running on, you really have to develop the GUI and user experience for each version of the application separately.
If you analyze what your application is going to do and figure that there is a significant portion of code/logic that could be shared between platforms then write that portion in a portable way in a language that is available on both systems. C, C++, Java, Python, Ruby, etc. If there isn't a significant portion, ie most of the code is going to be for the GUI then there is less of a case for sharing any code at all.
In the case that there is a significant portion of common code, I would suggest looking into Python and Ruby as implementation languages as there are Cocoa bindings for those languages on the Mac and on Windows with the use of IronPython and IronRuby you could use reuse that code in a .Net application as well.
