I have had a similar issue where the PyYAML package was installed by conda. There is another answer to use conda remove.
Instead I have got round this issue using conda update PyYAML, effectively using conda to update the dependency which pip is trying itself to update.