Java has the reputation of being slow because it was slow. The first versions of Java had either no or rather poor Just In Time compilation. This meant that the code, although bytecode, was being interpreted, so even for the simplest operations (like adding two integers) the machine had to do all sorts of compares and pointer dereferences and function calls. The JIT compiler has been ever-improving; now it's at the point where if I write C++ code carelessly and Java code carelessly, Java will sometimes outperform C++ because the JIT compiler realizes that I've got some unnecessary pointer dereferencing and will take care of it for me.
If you want to see just how big a difference JIT compilation makes, check out the interpreted vs. non-interpreted benchmarks at the Computer Languages Benchmark Game. (Pidigits uses an external library to do all the computations, so that benchmark doesn't change; the others show a 6-16x speedup!)
So, that's the main reason. There are a variety of other, lesser reasons that did not help matters: originally, Java startup time was slow (now fixed); web apps in Java take a long time to download (much less true now with widely accessible broadband, and with large things like movies being expected); the UI Swing was not (and still is not) written with performance in mind so it is much less snappy than equivalents in e.g. C++.