After doing a lot of things, I upgraded pip, setuptools and virtualenv.
python -m pip install -U pip
pip install -U setuptools
pip install -U virtualenv
I did steps 1, 2 in my virtual environment as well as globally.
Next, I installed the package through pip and it worked.