Trouble Installing Scrapy on 10.9 Mavericks

问题:

I'm trying to install Scrapy using my homebrew python build. Despite having everything up to date, installed and in my path, I cant use it.

Here's my order of operations:

check brew:

$ brew doctor Your system is ready to brew. 

check python & pip:

$ which python /usr/local/bin/python $ python --version Python 2.7.6 $ which pip /usr/local/bin/pip $ pip -- version pip 1.5.4 from /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7) 

(is this normal? this looks like there's two copies, possibly conflicting )

ive already install scrapy but when i try again i get:

$ pip install scrapy Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy) Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy) Cleaning up... 

this is weird to me because i expect my modules to be installed in /usr/local/share/python NOT in /usr/local/lib/python2.7/site-packages sure enough, there is no scrapy in /usr/local/lib/python2.7/site-packages but there is in /usr/local/share/python.

ive added /usr/local/share/python/scrapyto my path as discussed here: Scrapy installation on OSX Lion

but which scrapy returns blank.

the most interesting part is when i try to 'update' scrapy using pip install --upgrade scrapy :

~ ∮ pip install --upgrade scrapy Requirement already up-to-date: scrapy in /usr/local/lib/python2.7/site-packages Requirement already up-to-date: Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already up-to-date: w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already up-to-date: queuelib in /usr/local/lib/python2.7/site-packages (from scrapy) Downloading/unpacking lxml from https://pypi.python.org/packages/source/l/lxml/lxml-3.3.2.tar.gz#md5=a3ea7bf74b718ecb46d9fd5198eec92d (from scrapy)   Downloading lxml-3.3.2.tar.gz (3.5MB): 3.5MB downloaded   Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/lxml/setup.py) egg_info for package lxml     /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'       warnings.warn(msg)     Building lxml version 3.3.2.     Building without Cython.     Using build configuration of libxslt 1.1.28      warning: no previously-included files found matching '*.py' Downloading/unpacking pyOpenSSL from https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.14.tar.gz#md5=8579ff3a1d858858acfba5f046a4ddf7 (from scrapy)   Downloading pyOpenSSL-0.14.tar.gz (128kB): 128kB downloaded   Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/pyOpenSSL/setup.py) egg_info for package pyOpenSSL      warning: no previously-included files matching '*.pyc' found anywhere in distribution     no previously-included directories found matching 'doc/_build' Requirement already up-to-date: cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already up-to-date: six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy) Requirement already up-to-date: zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy) Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->scrapy)   Downloading cryptography-0.2.1.tar.gz (13.8MB): 13.8MB downloaded   Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py) egg_info for package cryptography     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing `libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing `libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing `libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing `libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found     Package libffi was not found in the pkg-config search path.     Perhaps you should add the directory containing `libffi.pc'     to the PKG_CONFIG_PATH environment variable     No package 'libffi' found      Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg     Searching for pycparser     Reading https://pypi.python.org/simple/pycparser/     Best match: pycparser 2.10     Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f     Processing pycparser-2.10.tar.gz     Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg     Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C     zip_safe flag not set; analyzing archive contents...      Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg      Traceback (most recent call last):       File "<string>", line 17, in <module>       File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>         "build": cffi_build,       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup         dist.run_commands()       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands         self.run_command(cmd)       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command         cmd_obj.run()       File "<string>", line 15, in replacement_run       File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources       File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run       File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults       File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command         cmd_obj.ensure_finalized()       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized         self.finalize_options()       File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options         ('force', 'force'))       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options         src_cmd_obj.ensure_finalized()       File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized         self.finalize_options()       File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options         from cryptography.hazmat.primitives import constant_time, padding       File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>         import six     ImportError: No module named six     Complete output from command python setup.py egg_info:     Package libffi was not found in the pkg-config search path.  Perhaps you should add the directory containing `libffi.pc'  to the PKG_CONFIG_PATH environment variable  No package 'libffi' found  Package libffi was not found in the pkg-config search path.  Perhaps you should add the directory containing `libffi.pc'  to the PKG_CONFIG_PATH environment variable  No package 'libffi' found  Package libffi was not found in the pkg-config search path.  Perhaps you should add the directory containing `libffi.pc'  to the PKG_CONFIG_PATH environment variable  No package 'libffi' found  Package libffi was not found in the pkg-config search path.  Perhaps you should add the directory containing `libffi.pc'  to the PKG_CONFIG_PATH environment variable  No package 'libffi' found  Package libffi was not found in the pkg-config search path.  Perhaps you should add the directory containing `libffi.pc'  to the PKG_CONFIG_PATH environment variable  No package 'libffi' found    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg  Searching for pycparser  Reading https://pypi.python.org/simple/pycparser/  Best match: pycparser 2.10  Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f  Processing pycparser-2.10.tar.gz  Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg  Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C  zip_safe flag not set; analyzing archive contents...    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg  running egg_info  creating pip-egg-info/cryptography.egg-info  writing requirements to pip-egg-info/cryptography.egg-info/requires.txt  writing pip-egg-info/cryptography.egg-info/PKG-INFO  writing top-level names to pip-egg-info/cryptography.egg-info/top_level.txt  writing dependency_links to pip-egg-info/cryptography.egg-info/dependency_links.txt  writing manifest file 'pip-egg-info/cryptography.egg-info/SOURCES.txt'  warning: manifest_maker: standard file '-c' not found    Traceback (most recent call last):    File "<string>", line 17, in <module>    File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>      "build": cffi_build,    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup      dist.run_commands()    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands      self.run_command(cmd)    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command      cmd_obj.run()    File "<string>", line 15, in replacement_run    File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources    File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run    File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults    File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command      cmd_obj.ensure_finalized()    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized      self.finalize_options()    File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options      ('force', 'force'))    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options      src_cmd_obj.ensure_finalized()    File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized      self.finalize_options()    File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options      from cryptography.hazmat.primitives import constant_time, padding    File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>      import six  ImportError: No module named six  ---------------------------------------- Cleaning up... Command python setup.py egg_info failed with error code 1 in /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_myAccount/cryptography Storing debug log for failure in /Users/myAccount/.pip/pip.log ~ ∮  

I've tried remedying some of these such as, pip install six and tried googling this egg_info failed with error code 1. most solutions involve not having setuptools installed, yet:

$ pip install setuptools Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg Cleaning up... 

it looks like my brew install of python and incorrect linking/multiple versions? of pip are causing modules to be installed in one place while my python is somewhere else (this explains why >>> import scrapy fails but not why $ scrapy fails ).

a similar post about broken python modules suggests running xcode-select --install which I've confirmed are up to date. Can't install PIL after Mac OS X 10.9

maybe its virtualenv time?

thanks in advance!

回答1:

Here's what worked for me:

1. Since you're using Homebrew as your package manager, the the programs installed through it will be set to /usr/local/bin rather than the standard /usr/bin path setting for out of the box applications. You'll want to ensure that /usr/local/bin is at the top of the file in /etc/paths. The file should be locked, so you may need to save changes in the vi editor with the following:

   :w !sudo tee % 

2. $ which python should return /usr/local/bin/python, else:

   `$ brew unlink python` `$ brew install python` `$ brew linkapps` 

3. $ which pip should return /usr/local/bin/pip, else

   `$ brew install pip` 

4. Then...

   `$ sudo pip uninstall Scrapy` `$ pip install Scrapy` `$ pip install --upgrade Scrapy` `$ pip install --upgrade lxml` `$ scrapy startproject tutorial` 

回答2:

I had a similar issue installing virtualenv: Can't pip install virtualenv on Mac 10.9 Mavericks with brewed python

the solution there works here as well, although I still do not understand WHY the problem occurred in the first place and why this solution works. But it does.

uninstall and reinstall scrapy:

$pip uninstall scrapy $pip install scrapy  

This removes modules from /usr/local/lib/python2.7/site-packages and installs them in /usr/local/bin/

The problem is that according to the homebrew python docs, pip is suppose to install in /site-packages and they should work. Not only does it not work with them in /site-packages but pip defaults to installing in /usr/local/bin/.

This workaround works, although I would really appreciate an explanation as to why it works and why this was a problem in the first place. Then perhaps these two questions can be merged?

回答3:

None of the above worked for me in OS X 10.10 Yosemite.

But I found a solution here.

It's using pkg-config:

brew install pkg-config libffi export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/ pip install cffi 

You will have to change the path according to your libffi version.

回答4:

You need to give your installation environment access to the Crypto library bundled with OpenSSL. Here is the solution that worked for me.

env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include"

Remember to change this to the location of your updated OpenSSL package.

回答5:

The other posts didn't work for me, but this did:

sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install scrapy 

文章来源: Trouble Installing Scrapy on 10.9 Mavericks