NOTE: I\'m not working at the company where I worked when I posted this question, therefore, even though some great answers might come in, I won\'t act
For everyone struggling with this problem here's the solution:
Android MediaPlayer takes long time to prepare and buffer
EDIT : The previous solution is not very complete because it is possible sometimes to hear the player 'stutter' when it pauses and resumes.
A 100% java answer that is slightly more elegant involves using a MediaCodec instance to turn the mp3 into PCM data to feed to an AudioTrack instance.
I have posted full source code and explanation here : http://www.piterwilson.com/blog/2014/03/15/mediacodec-mediaextractor-and-audiotrack-to-the-rescue/