When I try to upgrade my matplotlib using pip, it outputs:
Downloading/unpacking matplotlib from https://pypi.python
On Ubuntu 14 server, you also need to install libxft-dev
sudo apt-get install libfreetype6-dev libxft-dev