Is there a not so ugly way of treat the close() exception to close both streams then:
InputStream in = new FileInputStream(inputFileName);
Guava has very nice IO APIs that eliminate the need for this. For instance, your example would be:
Files.copy(new File(inputFileName), new File(outputFileName));
More generally, it uses the concept of InputSuppliers and OutputSuppliers to allow the InputStreams and OutputStreams to be created within its utility methods, allowing it full control over them so it can handle closing properly.
Additionally, it has Closeables.closeQuietly(Closeable) which is basically the type of method most of the answers have suggested.
The IO stuff in it is still in beta and subject to change, but it's worth checking out and even using, depending on what it is you're working on.