When I try to upgrade my matplotlib using pip, it outputs:
Downloading/unpacking matplotlib from https://pypi.python
If you re running Ubuntu server 14.04 u should add this font dependency
sudo apt-get install libxft-dev
Source