When someone clicks to watch a video on my site (mp4 in an Html5 video tag) - the browser buffers a lot of it before showing it. This is not needed - the video is less than
There's been a lot of discussion in the comments regarding whether this can be done, so I'll provide a Media Source-specific answer here.
Media Source Extensions, or MSE, are a new (and not yet widely-supported) set of tools to help you control buffering and streaming with HTML5 videos. From the W3C abstract:
This specification extends HTMLMediaElement to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams.
I'll specifically refer you to the SourceBuffer Object, which has information on how audio & video track buffering is handled.
Support for MSE varies by browser and format (source):
Support for Firefox can be turned on by the user in about:config (source). Support has been in the works for some time.
There is much more that needs to be implemented in order to make use of this effectively, including video file clustering. I would recommend reading this 4-part series that goes step by step into how to create a fully functional HTML5 video player utilizing everything mentioned above.