I am working on an online TV service. One of the goals is for the video to be played without any additional browser plug-ins (except for Flash).
I decided to use MP4
Just looking at 2nd para of your question ("The videos are transcoded from ASF on a server by ffmpeg."), you mentioned you are using ffmpeg to transcode videos on server.
Use qt-faststart or MP4Box to place MOOV atom in the beginning of the file. (also you make sure that using H264 Video & AAC Audio codec for universal support)
Hope this helped you.