You can use a specific environment variable PIP_DOWNLOAD_CACHE and make it point to a directory where your packages will be stored. If they are to be installed again, they will be taken from this directory.
There seems to be also an additional option for PIP pip --download-cache which ought to do something similar, but I have never tried it myself. For your example, to avoid re-downloading matplotlib every time, you would do the following:
pip install --download-cache /path/to/pip/cache matplotlib
Does that answer your question?