When I try to upgrade my matplotlib using pip, it outputs:
Downloading/unpacking matplotlib from https://pypi.python
I had the same issues trying to install matplotlib on Python 3 using pip3, and it seems that this problem is related to a bare-bones installation of Python 3, and doing a:
sudo apt-get build-dep matplotlib
followed by
sudo pip3 install matplotlib
is probably a better solution than selectively installing only the libraries related to matplotlib.