This is a popular question that is getting more attention than it probably should. In order to stop the spread of misinformation please read
https works as follows: First a 4-way handshake is performed (at least if i remember correctly it was 4way), here client and server agree on the symmetric encryption algorithm used later on and exchange certificates (containing public keys).
They exchange a session (key for the symmetric enc later) using publickey crypto.
Now they send messages encrypted with the session key and some encryption algorithm (3des, aes, rc4, rc5, etc). Since symmetric encryptions are not that expensive operations the differences in download time are not that big.
The fact that you have less waiting time is because you probably have less traffic on http port or less traffic at the time you did the https request compared to the http requests.
So to optimize performance you should use as few https connections as possible since the handshake is a relative expensive procedure.