For most people's experience interacting with it - Java is slow. We've all seen that coffee cup spinning around on our browser before some applet comes up. It takes a while to spin up the JVM and download the applet binaries, and that impacts the user experience in a way that is noticed.
It doesn't help that the slow JVM spin-up and applet download time is conspicuously branded with a Java coffee cup, so people associate the wait with Java. When Flash takes a long time to load, the branding of the "loading" message is specified by the Flash developer, so people don't blame Flash technology as a whole.
All of this has nothing to do with Java's performance on a server, or in the many other ways that Java gets used outside the browser. But it's what people see, and what non-Java developers remember when they think about Java.