问题
I'm trying to install Scrapy using my homebrew python build. Despite having everything up to date, installed and in my path, I cant use it.
Here's my order of operations:
check brew:
$ brew doctor
Your system is ready to brew.
check python & pip:
$ which python
/usr/local/bin/python
$ python --version
Python 2.7.6
$ which pip
/usr/local/bin/pip
$ pip -- version
pip 1.5.4 from /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7)
(is this normal? this looks like there's two copies, possibly conflicting )
ive already install scrapy but when i try again i get:
$ pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy)
Cleaning up...
this is weird to me because i expect my modules to be installed in /usr/local/share/python NOT in /usr/local/lib/python2.7/site-packages
sure enough, there is no scrapy in /usr/local/lib/python2.7/site-packages but there is in /usr/local/share/python.
ive added /usr/local/share/python/scrapyto my path as discussed here: Scrapy installation on OSX Lion
but which scrapy returns blank.
the most interesting part is when i try to 'update' scrapy using pip install --upgrade scrapy :
~ ∮ pip install --upgrade scrapy
Requirement already up-to-date: scrapy in /usr/local/lib/python2.7/site-packages
Requirement already up-to-date: Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Downloading/unpacking lxml from https://pypi.python.org/packages/source/l/lxml/lxml-3.3.2.tar.gz#md5=a3ea7bf74b718ecb46d9fd5198eec92d (from scrapy)
Downloading lxml-3.3.2.tar.gz (3.5MB): 3.5MB downloaded
Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/lxml/setup.py) egg_info for package lxml
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
Building lxml version 3.3.2.
Building without Cython.
Using build configuration of libxslt 1.1.28
warning: no previously-included files found matching '*.py'
Downloading/unpacking pyOpenSSL from https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.14.tar.gz#md5=8579ff3a1d858858acfba5f046a4ddf7 (from scrapy)
Downloading pyOpenSSL-0.14.tar.gz (128kB): 128kB downloaded
Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/pyOpenSSL/setup.py) egg_info for package pyOpenSSL
warning: no previously-included files matching '*.pyc' found anywhere in distribution
no previously-included directories found matching 'doc/_build'
Requirement already up-to-date: cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->scrapy)
Downloading cryptography-0.2.1.tar.gz (13.8MB): 13.8MB downloaded
Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py) egg_info for package cryptography
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg
Searching for pycparser
Reading https://pypi.python.org/simple/pycparser/
Best match: pycparser 2.10
Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f
Processing pycparser-2.10.tar.gz
Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg
Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C
zip_safe flag not set; analyzing archive contents...
Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>
"build": cffi_build,
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "<string>", line 15, in replacement_run
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options
('force', 'force'))
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options
src_cmd_obj.ensure_finalized()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options
from cryptography.hazmat.primitives import constant_time, padding
File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>
import six
ImportError: No module named six
Complete output from command python setup.py egg_info:
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg
Searching for pycparser
Reading https://pypi.python.org/simple/pycparser/
Best match: pycparser 2.10
Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f
Processing pycparser-2.10.tar.gz
Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg
Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C
zip_safe flag not set; analyzing archive contents...
Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg
running egg_info
creating pip-egg-info/cryptography.egg-info
writing requirements to pip-egg-info/cryptography.egg-info/requires.txt
writing pip-egg-info/cryptography.egg-info/PKG-INFO
writing top-level names to pip-egg-info/cryptography.egg-info/top_level.txt
writing dependency_links to pip-egg-info/cryptography.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/cryptography.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>
"build": cffi_build,
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "<string>", line 15, in replacement_run
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options
('force', 'force'))
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options
src_cmd_obj.ensure_finalized()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options
from cryptography.hazmat.primitives import constant_time, padding
File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>
import six
ImportError: No module named six
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_myAccount/cryptography
Storing debug log for failure in /Users/myAccount/.pip/pip.log
~ ∮
I've tried remedying some of these such as, pip install six and tried googling this egg_info failed with error code 1.
most solutions involve not having setuptools installed, yet:
$ pip install setuptools
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg
Cleaning up...
it looks like my brew install of python and incorrect linking/multiple versions? of pip are causing modules to be installed in one place while my python is somewhere else (this explains why >>> import scrapy fails but not why $ scrapy fails ).
a similar post about broken python modules suggests running xcode-select --install which I've confirmed are up to date. Can't install PIL after Mac OS X 10.9
maybe its virtualenv time?
thanks in advance!
回答1:
Here's what worked for me:
Since you're using Homebrew as your package manager, the the programs installed through it will be set to
/usr/local/binrather than the standard/usr/binpath setting for out of the box applications. You'll want to ensure that/usr/local/binis at the top of the file in/etc/paths. The file should be locked, so you may need to save changes in the vi editor with the following::w !sudo tee %$ which pythonshould return/usr/local/bin/python, else:`$ brew unlink python` `$ brew install python` `$ brew linkapps`$ which pipshould return/usr/local/bin/pip, else`$ brew install pip`Then...
`$ sudo pip uninstall Scrapy` `$ pip install Scrapy` `$ pip install --upgrade Scrapy` `$ pip install --upgrade lxml` `$ scrapy startproject tutorial`
回答2:
I had a similar issue installing virtualenv: Can't pip install virtualenv on Mac 10.9 Mavericks with brewed python
the solution there works here as well, although I still do not understand WHY the problem occurred in the first place and why this solution works. But it does.
uninstall and reinstall scrapy:
$pip uninstall scrapy
$pip install scrapy
This removes modules from /usr/local/lib/python2.7/site-packages
and installs them in /usr/local/bin/
The problem is that according to the homebrew python docs, pip is suppose to install in /site-packages and they should work.
Not only does it not work with them in /site-packages but pip defaults to installing in /usr/local/bin/.
This workaround works, although I would really appreciate an explanation as to why it works and why this was a problem in the first place. Then perhaps these two questions can be merged?
回答3:
None of the above worked for me in OS X 10.10 Yosemite.
But I found a solution here.
It's using pkg-config:
brew install pkg-config libffi
export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/
pip install cffi
You will have to change the path according to your libffi version.
回答4:
You need to give your installation environment access to the Crypto library bundled with OpenSSL. Here is the solution that worked for me.
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include"
Remember to change this to the location of your updated OpenSSL package.
回答5:
The other posts didn't work for me, but this did:
sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install scrapy
来源：https://stackoverflow.com/questions/22106687/trouble-installing-scrapy-on-10-9-mavericks