Another answer along the lines of David's...you can use jQuery to clear the SRC attribute of the iFrame.
$('iframe#targetID').attr('src','');
I'm using this with a Vimeo video and a lightbox effect. When the lightbox is triggered again, I add the video URL back to the iFrame SRC before showing it.